2021-11-20T05:45:31.661 INFO:root:teuthology version: 1.1.0-6cfddd5e 2021-11-20T05:45:31.664 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514834 --owner scheduled_yuriw@teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi -- /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514834/orig.config.yaml 2021-11-20T05:45:31.732 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-11-20T05:45:31.809 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514834 branch: wip-yuri2-testing-2021-11-19-1714-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} email: ceph-qa@ceph.io first_in_suite: false job_id: '6514834' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri2-testing-2021-11-19-1714-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 15m rados osd op timeout: 15m global: mon pg warn min per osd: 0 mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri2-testing-2021-11-19-1714-pacific sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa owner: scheduled_yuriw@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri2-testing-2021-11-19-1714-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa targets: smithi139.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDX/gOOMNH79j8Nrq/rHi86pfDdC6mN8mEpmMq2U6A8T7N6qBHRYRFE/+BBQJb8BAfheJJSWrBdnomJx+pP9vucEMpbodi+/M60JwcrkbllEEl03AgB5uAQzzNH9aRRFmDOMCn9EiZqJ/KS21KNqH+DsYcMt1Dzdx+i5gH+nUAlk+pJ7fpQ5WaFdhO1AvYGmXkEyKnKDXwba6jsHk4OwjfcunS17+uKfnGUdMcqhT9rFpE48qA0VKS8QvMpbgpiq/r6tNOD21Cqt12RswhpZc6e6OYjDXJ5ndwcz1rUHRGannMnn71PVYxH3RWD+KLio0gPQgufLMpD4DCkXT+i4ZI4KNbN56fdyR8CfQrNcISPuiDegrULR3+uYzoLoFijK6f1TIW6ZxYOuy5jEnZUfz5mp55H/L5WwXTorAvqdxg8v4yPaVKB4yEQ4Z2QG3OFpXYSng6vdu4dhGdNs4uyJgnoUP68QCBu+CQ4Ejz0ub01K0iz1xYlxcr2edy1BW6vMj8= smithi178.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDs70d0MsnQ8YemPH6h36UVsNhIHHmYotC0jDJmSlm65Na2P1NXiiVcvzQZW9ft4PU/1jOBk9dL0cCTgrJ2OM0FLDdi8/pZiAA8Mvs4Ql5ztmSMGUmtMZOh92t0CdWpEDUcVebTiSpWShTQHA3bTmpTi6PxiULcZYVpjB8qvcRZQIs1dnnWLipGMXWle8tTrHvq8trYEnYJul39HJGSYxBKBoV4tsZ7waXSBseUFvoiFm3QrCmlR45W7X469wuCr4iC1wRIgCXcq7ndqQMrSOD140bb/M1GUnXrvnj4Sbc9Pc3BE5WeyfHSzpAoLKyXLzfkCWxI7DbmQBU4svMxB364V6Bt79Eq9S40kZFRHBn4ceaMbajJS6PWg491gNnctjLPjWj2wlagTOYbrR7vpKIM6YpvH/8G0hazT+QZWqkGRW+f6O61t7tvaY5pFBIJVv9vK0EeRD5Njrbyt6QYpPi5EoxPNLH+ffSCi0OPDrw2hG3YS/i3JU60rDk0JltXzk= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: exclude_packages: - ceph-volume tag: v16.2.4 - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - ceph --format=json mds versions | jq -e ". | add == 4" - fs.pre_upgrade_save: null - kclient: null - print: '**** done kclient client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology_branch: master teuthology_sha1: 6cfddd5e1c7d5ede1367cf3087c21af44eb88356 timestamp: 2021-11-20_04:32:48 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; ceph fs dump; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.20261 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2021-11-20T05:45:31.809 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa; will attempt to use it 2021-11-20T05:45:31.810 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks 2021-11-20T05:45:31.810 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-11-20T05:45:31.812 INFO:teuthology.task.internal:Checking packages... 2021-11-20T05:45:31.833 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa' 2021-11-20T05:45:31.834 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-11-20T05:45:31.834 INFO:teuthology.packaging:ref: None 2021-11-20T05:45:31.835 INFO:teuthology.packaging:tag: None 2021-11-20T05:45:31.835 INFO:teuthology.packaging:branch: wip-yuri2-testing-2021-11-19-1714-pacific 2021-11-20T05:45:31.835 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T05:45:31.836 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2021-11-19-1714-pacific 2021-11-20T05:45:32.047 INFO:teuthology.task.internal:Found packages for ceph version 16.2.6-633.g9a97ad1b 2021-11-20T05:45:32.048 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-11-20T05:45:32.059 INFO:teuthology.task.internal:no buildpackages task found 2021-11-20T05:45:32.059 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-11-20T05:45:32.098 INFO:teuthology.task.internal:Saving configuration 2021-11-20T05:45:32.118 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-11-20T05:45:32.132 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-20T05:45:32.159 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514834', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-11-20 05:40:05.029692', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIOJ6jnbwRYIpuv6OgF3qYsUGD4LsJ3Zd9ixx7KEkE39/x7Lsh/2odFEKQdyvmCXvd1lBUcUKz513RwiKxupFqU='} 2021-11-20T05:45:32.182 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514834', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-11-20 05:40:05.028426', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLeFHv984BRiLT2yTFAq7luaUVxcMDhjJGnDxTiFEI5Y1Kruzjy5SrE2IuQ2g7D+nXuO2+9sifta1/5rGWtyqSQ='} 2021-11-20T05:45:32.183 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-11-20T05:45:32.194 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2021-11-20T05:45:32.195 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2021-11-20T05:45:32.195 INFO:teuthology.run_tasks:Running task console_log... 2021-11-20T05:45:32.283 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa864a29598>, signals=[15]) 2021-11-20T05:45:32.284 INFO:teuthology.run_tasks:Running task internal.connect... 2021-11-20T05:45:32.296 INFO:teuthology.task.internal:Opening connections... 2021-11-20T05:45:32.296 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi139.front.sepia.ceph.com 2021-11-20T05:45:32.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:45:32.372 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2021-11-20T05:45:32.373 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:45:32.441 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-11-20T05:45:32.452 DEBUG:teuthology.orchestra.run.smithi139:> uname -m 2021-11-20T05:45:32.469 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2021-11-20T05:45:32.470 DEBUG:teuthology.orchestra.run.smithi139:> cat /etc/os-release 2021-11-20T05:45:32.532 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="CentOS Stream" 2021-11-20T05:45:32.532 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="8" 2021-11-20T05:45:32.533 INFO:teuthology.orchestra.run.smithi139.stdout:ID="centos" 2021-11-20T05:45:32.533 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="rhel fedora" 2021-11-20T05:45:32.535 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="8" 2021-11-20T05:45:32.535 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el8" 2021-11-20T05:45:32.535 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-20T05:45:32.536 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31" 2021-11-20T05:45:32.536 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-20T05:45:32.537 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://centos.org/" 2021-11-20T05:45:32.538 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-20T05:45:32.538 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-20T05:45:32.539 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-20T05:45:32.540 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2021-11-20T05:45:32.562 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2021-11-20T05:45:32.582 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2021-11-20T05:45:32.583 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2021-11-20T05:45:32.641 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="CentOS Stream" 2021-11-20T05:45:32.641 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="8" 2021-11-20T05:45:32.641 INFO:teuthology.orchestra.run.smithi178.stdout:ID="centos" 2021-11-20T05:45:32.641 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="rhel fedora" 2021-11-20T05:45:32.642 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="8" 2021-11-20T05:45:32.642 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el8" 2021-11-20T05:45:32.642 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-20T05:45:32.643 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31" 2021-11-20T05:45:32.643 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-20T05:45:32.643 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://centos.org/" 2021-11-20T05:45:32.643 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-20T05:45:32.643 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-20T05:45:32.644 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-20T05:45:32.644 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2021-11-20T05:45:32.664 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-11-20T05:45:32.683 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-11-20T05:45:32.694 INFO:teuthology.task.internal:Checking for old test directory... 2021-11-20T05:45:32.694 DEBUG:teuthology.orchestra.run.smithi139:> test '!' -e /home/ubuntu/cephtest 2021-11-20T05:45:32.698 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2021-11-20T05:45:32.716 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-11-20T05:45:32.727 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-11-20T05:45:32.728 DEBUG:teuthology.orchestra.run.smithi139:> test -z $(ls -A /var/lib/ceph) 2021-11-20T05:45:32.755 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2021-11-20T05:45:32.780 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-11-20T05:45:32.836 INFO:teuthology.run_tasks:Running task kernel... 2021-11-20T05:45:32.863 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2021-11-20T05:45:32.864 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-11-20T05:45:32.864 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2021-11-20T05:45:32.865 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2021-11-20T05:45:32.882 INFO:teuthology.orchestra.run.smithi139.stdout:4.18.0-301.1.el8.x86_64 2021-11-20T05:45:32.883 INFO:teuthology.task.kernel:Running kernel on smithi139: 4.18.0-301.1.el8.x86_64 2021-11-20T05:45:32.883 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum install -y kernel 2021-11-20T05:45:32.884 INFO:teuthology.orchestra.run.smithi178.stdout:4.18.0-301.1.el8.x86_64 2021-11-20T05:45:32.885 INFO:teuthology.task.kernel:Running kernel on smithi178: 4.18.0-301.1.el8.x86_64 2021-11-20T05:45:32.885 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum install -y kernel 2021-11-20T05:45:35.476 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - AppStream 53 MB/s | 17 MB 00:00 2021-11-20T05:45:35.763 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-8 - AppStream 30 MB/s | 17 MB 00:00 2021-11-20T05:45:38.635 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - Base 62 MB/s | 15 MB 00:00 2021-11-20T05:45:38.914 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-8 - Base 62 MB/s | 15 MB 00:00 2021-11-20T05:45:40.841 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - Extras 208 kB/s | 16 kB 00:00 2021-11-20T05:45:41.131 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-8 - Extras 236 kB/s | 16 kB 00:00 2021-11-20T05:45:42.103 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 12 MB/s | 11 MB 00:00 2021-11-20T05:45:42.401 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 11 MB 00:01 2021-11-20T05:45:44.546 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2021-11-20T05:45:44.818 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2021-11-20T05:45:46.519 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-4.18.0-240.22.1.el8_3.x86_64 is already installed. 2021-11-20T05:45:46.520 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-4.18.0-301.1.el8.x86_64 is already installed. 2021-11-20T05:45:46.520 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-11-20T05:45:46.847 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:45:46.849 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:45:46.849 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2021-11-20T05:45:46.849 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:45:46.849 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:45:46.850 INFO:teuthology.orchestra.run.smithi178.stdout: kernel x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 7.0 M 2021-11-20T05:45:46.850 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-core x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 38 M 2021-11-20T05:45:46.850 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 30 M 2021-11-20T05:45:46.850 INFO:teuthology.orchestra.run.smithi178.stdout:Removing: 2021-11-20T05:45:46.850 INFO:teuthology.orchestra.run.smithi178.stdout: kernel x86_64 4.18.0-240.el8 @anaconda 0 2021-11-20T05:45:46.851 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-core x86_64 4.18.0-240.el8 @anaconda 62 M 2021-11-20T05:45:46.851 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules x86_64 4.18.0-240.el8 @anaconda 21 M 2021-11-20T05:45:46.851 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:45:46.851 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:45:46.852 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:45:46.852 INFO:teuthology.orchestra.run.smithi178.stdout:Install 3 Packages 2021-11-20T05:45:46.852 INFO:teuthology.orchestra.run.smithi178.stdout:Remove 3 Packages 2021-11-20T05:45:46.852 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:45:46.853 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 74 M 2021-11-20T05:45:46.853 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:45:46.868 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-4.18.0-240.22.1.el8_3.x86_64 is already installed. 2021-11-20T05:45:46.868 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-4.18.0-301.1.el8.x86_64 is already installed. 2021-11-20T05:45:46.869 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-11-20T05:45:47.085 INFO:teuthology.orchestra.run.smithi178.stdout:(1/3): kernel-4.18.0-348.2.1.el8_5.x86_64.rpm 30 MB/s | 7.0 MB 00:00 2021-11-20T05:45:47.174 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:45:47.175 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:45:47.176 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-11-20T05:45:47.176 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:45:47.176 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:45:47.176 INFO:teuthology.orchestra.run.smithi139.stdout: kernel x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 7.0 M 2021-11-20T05:45:47.176 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-core x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 38 M 2021-11-20T05:45:47.177 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 30 M 2021-11-20T05:45:47.177 INFO:teuthology.orchestra.run.smithi139.stdout:Removing: 2021-11-20T05:45:47.177 INFO:teuthology.orchestra.run.smithi139.stdout: kernel x86_64 4.18.0-240.el8 @anaconda 0 2021-11-20T05:45:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-core x86_64 4.18.0-240.el8 @anaconda 62 M 2021-11-20T05:45:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules x86_64 4.18.0-240.el8 @anaconda 21 M 2021-11-20T05:45:47.178 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:45:47.178 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:45:47.179 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:45:47.179 INFO:teuthology.orchestra.run.smithi139.stdout:Install 3 Packages 2021-11-20T05:45:47.179 INFO:teuthology.orchestra.run.smithi139.stdout:Remove 3 Packages 2021-11-20T05:45:47.179 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:45:47.180 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 74 M 2021-11-20T05:45:47.180 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:45:47.825 INFO:teuthology.orchestra.run.smithi139.stdout:(1/3): kernel-4.18.0-348.2.1.el8_5.x86_64.rpm 11 MB/s | 7.0 MB 00:00 2021-11-20T05:45:47.928 INFO:teuthology.orchestra.run.smithi178.stdout:(2/3): kernel-modules-4.18.0-348.2.1.el8_5.x86_ 28 MB/s | 30 MB 00:01 2021-11-20T05:45:48.395 INFO:teuthology.orchestra.run.smithi178.stdout:(3/3): kernel-core-4.18.0-348.2.1.el8_5.x86_64. 24 MB/s | 38 MB 00:01 2021-11-20T05:45:48.401 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:45:48.402 INFO:teuthology.orchestra.run.smithi178.stdout:Total 48 MB/s | 74 MB 00:01 2021-11-20T05:45:48.531 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:45:48.574 INFO:teuthology.orchestra.run.smithi139.stdout:(2/3): kernel-modules-4.18.0-348.2.1.el8_5.x86_ 21 MB/s | 30 MB 00:01 2021-11-20T05:45:48.917 INFO:teuthology.orchestra.run.smithi139.stdout:(3/3): kernel-core-4.18.0-348.2.1.el8_5.x86_64. 22 MB/s | 38 MB 00:01 2021-11-20T05:45:48.922 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:45:48.922 INFO:teuthology.orchestra.run.smithi139.stdout:Total 43 MB/s | 74 MB 00:01 2021-11-20T05:45:49.056 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:45:49.292 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:45:49.293 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:45:49.689 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:45:49.689 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:45:50.405 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:45:50.410 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:45:50.820 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:45:50.835 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:45:53.071 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:45:53.614 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T05:45:53.792 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:45:54.281 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T05:45:57.808 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T05:45:58.197 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T05:45:58.386 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T05:45:58.797 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T05:46:01.258 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T05:46:01.467 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T05:46:01.468 INFO:teuthology.orchestra.run.smithi178.stdout: Erasing : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T05:46:01.676 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T05:46:01.677 INFO:teuthology.orchestra.run.smithi178.stdout: Erasing : kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T05:46:01.862 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T05:46:02.117 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T05:46:02.117 INFO:teuthology.orchestra.run.smithi139.stdout: Erasing : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T05:46:02.332 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T05:46:02.332 INFO:teuthology.orchestra.run.smithi139.stdout: Erasing : kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T05:46:03.853 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T05:46:04.061 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:46:04.071 INFO:teuthology.orchestra.run.smithi178.stdout: Erasing : kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:46:04.387 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-20T05:46:04.576 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:46:04.586 INFO:teuthology.orchestra.run.smithi139.stdout: Erasing : kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:46:06.973 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:46:07.481 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:46:27.816 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 6/6 2021-11-20T05:46:28.148 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 6/6 2021-11-20T05:46:28.239 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:46:28.239 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T05:46:28.240 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-core-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T05:46:28.240 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T05:46:28.240 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T05:46:28.240 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-core-4.18.0-240.el8.x86_64 5/6 2021-11-20T05:46:28.537 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:46:28.537 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-20T05:46:28.538 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-core-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-20T05:46:28.538 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-20T05:46:28.538 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-20T05:46:28.538 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-core-4.18.0-240.el8.x86_64 5/6 2021-11-20T05:46:28.765 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-modules-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:46:28.766 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:46:28.766 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:46:28.766 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:28.766 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-core-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:28.767 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:28.767 INFO:teuthology.orchestra.run.smithi178.stdout:Removed: 2021-11-20T05:46:28.767 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-4.18.0-240.el8.x86_64 kernel-core-4.18.0-240.el8.x86_64 2021-11-20T05:46:28.767 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-4.18.0-240.el8.x86_64 2021-11-20T05:46:28.767 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:46:28.768 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:46:28.892 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel --last | head -n 1 2021-11-20T05:46:29.018 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-modules-4.18.0-240.el8.x86_64 6/6 2021-11-20T05:46:29.018 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:46:29.018 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:46:29.019 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:29.019 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-core-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:29.019 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:29.019 INFO:teuthology.orchestra.run.smithi139.stdout:Removed: 2021-11-20T05:46:29.019 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-4.18.0-240.el8.x86_64 kernel-core-4.18.0-240.el8.x86_64 2021-11-20T05:46:29.020 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-4.18.0-240.el8.x86_64 2021-11-20T05:46:29.020 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:46:29.021 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:46:29.058 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 05:46:01 AM UTC 2021-11-20T05:46:29.062 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:29.062 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-301.1.el8.x86_64 Expected: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:29.063 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-11-20T05:46:29.063 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-11-20T05:46:29.064 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2021-11-20T05:46:29.064 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi178.front.sepia.ceph.com, path=None, version=distro) 2021-11-20T05:46:29.065 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel --last | head -n 1 2021-11-20T05:46:29.144 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q kernel --last | head -n 1 2021-11-20T05:46:29.208 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 05:46:01 AM UTC 2021-11-20T05:46:29.210 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:29.210 DEBUG:teuthology.orchestra.run.smithi178:> sudo rpm -qi grub2-tools 2021-11-20T05:46:29.244 INFO:teuthology.orchestra.run.smithi178.stdout:Name : grub2-tools 2021-11-20T05:46:29.244 INFO:teuthology.orchestra.run.smithi178.stdout:Epoch : 1 2021-11-20T05:46:29.244 INFO:teuthology.orchestra.run.smithi178.stdout:Version : 2.02 2021-11-20T05:46:29.245 INFO:teuthology.orchestra.run.smithi178.stdout:Release : 99.el8 2021-11-20T05:46:29.245 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: x86_64 2021-11-20T05:46:29.245 INFO:teuthology.orchestra.run.smithi178.stdout:Install Date: Thu 13 May 2021 06:08:01 PM UTC 2021-11-20T05:46:29.245 INFO:teuthology.orchestra.run.smithi178.stdout:Group : System Environment/Base 2021-11-20T05:46:29.245 INFO:teuthology.orchestra.run.smithi178.stdout:Size : 9264704 2021-11-20T05:46:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:License : GPLv3+ 2021-11-20T05:46:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 09:11:55 PM UTC, Key ID 05b555b38483c65d 2021-11-20T05:46:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2021-11-20T05:46:29.246 INFO:teuthology.orchestra.run.smithi178.stdout:Build Date : Tue 02 Mar 2021 09:03:27 PM UTC 2021-11-20T05:46:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:Build Host : kbuilder.bsys.centos.org 2021-11-20T05:46:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:Relocations : (not relocatable) 2021-11-20T05:46:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:Packager : CentOS BuildSystem 2021-11-20T05:46:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:Vendor : CentOS 2021-11-20T05:46:29.247 INFO:teuthology.orchestra.run.smithi178.stdout:URL : http://www.gnu.org/software/grub/ 2021-11-20T05:46:29.248 INFO:teuthology.orchestra.run.smithi178.stdout:Summary : Support tools for GRUB. 2021-11-20T05:46:29.248 INFO:teuthology.orchestra.run.smithi178.stdout:Description : 2021-11-20T05:46:29.248 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:46:29.248 INFO:teuthology.orchestra.run.smithi178.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-11-20T05:46:29.249 INFO:teuthology.orchestra.run.smithi178.stdout:customizable bootloader with modular architecture. It supports a rich 2021-11-20T05:46:29.249 INFO:teuthology.orchestra.run.smithi178.stdout:variety of kernel formats, file systems, computer architectures and 2021-11-20T05:46:29.249 INFO:teuthology.orchestra.run.smithi178.stdout:hardware devices. 2021-11-20T05:46:29.249 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:46:29.250 INFO:teuthology.orchestra.run.smithi178.stdout:This subpackage provides tools for support of all platforms. 2021-11-20T05:46:29.250 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-11-20T05:46:29.250 INFO:teuthology.task.kernel:Updating grub on smithi178 to boot 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:29.250 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-11-20T05:46:29.346 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 05:46:01 AM UTC 2021-11-20T05:46:29.347 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:29.348 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-301.1.el8.x86_64 Expected: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:29.348 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-11-20T05:46:29.348 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-11-20T05:46:29.349 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2021-11-20T05:46:29.349 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi139.front.sepia.ceph.com, path=None, version=distro) 2021-11-20T05:46:29.349 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q kernel --last | head -n 1 2021-11-20T05:46:29.491 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Sat 20 Nov 2021 05:46:01 AM UTC 2021-11-20T05:46:29.493 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:29.493 DEBUG:teuthology.orchestra.run.smithi139:> sudo rpm -qi grub2-tools 2021-11-20T05:46:29.524 INFO:teuthology.orchestra.run.smithi139.stdout:Name : grub2-tools 2021-11-20T05:46:29.524 INFO:teuthology.orchestra.run.smithi139.stdout:Epoch : 1 2021-11-20T05:46:29.524 INFO:teuthology.orchestra.run.smithi139.stdout:Version : 2.02 2021-11-20T05:46:29.525 INFO:teuthology.orchestra.run.smithi139.stdout:Release : 99.el8 2021-11-20T05:46:29.525 INFO:teuthology.orchestra.run.smithi139.stdout:Architecture: x86_64 2021-11-20T05:46:29.525 INFO:teuthology.orchestra.run.smithi139.stdout:Install Date: Thu 13 May 2021 06:08:01 PM UTC 2021-11-20T05:46:29.525 INFO:teuthology.orchestra.run.smithi139.stdout:Group : System Environment/Base 2021-11-20T05:46:29.526 INFO:teuthology.orchestra.run.smithi139.stdout:Size : 9264704 2021-11-20T05:46:29.526 INFO:teuthology.orchestra.run.smithi139.stdout:License : GPLv3+ 2021-11-20T05:46:29.527 INFO:teuthology.orchestra.run.smithi139.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 09:11:55 PM UTC, Key ID 05b555b38483c65d 2021-11-20T05:46:29.527 INFO:teuthology.orchestra.run.smithi139.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2021-11-20T05:46:29.527 INFO:teuthology.orchestra.run.smithi139.stdout:Build Date : Tue 02 Mar 2021 09:03:27 PM UTC 2021-11-20T05:46:29.527 INFO:teuthology.orchestra.run.smithi139.stdout:Build Host : kbuilder.bsys.centos.org 2021-11-20T05:46:29.527 INFO:teuthology.orchestra.run.smithi139.stdout:Relocations : (not relocatable) 2021-11-20T05:46:29.528 INFO:teuthology.orchestra.run.smithi139.stdout:Packager : CentOS BuildSystem 2021-11-20T05:46:29.528 INFO:teuthology.orchestra.run.smithi139.stdout:Vendor : CentOS 2021-11-20T05:46:29.528 INFO:teuthology.orchestra.run.smithi139.stdout:URL : http://www.gnu.org/software/grub/ 2021-11-20T05:46:29.528 INFO:teuthology.orchestra.run.smithi139.stdout:Summary : Support tools for GRUB. 2021-11-20T05:46:29.529 INFO:teuthology.orchestra.run.smithi139.stdout:Description : 2021-11-20T05:46:29.529 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:46:29.529 INFO:teuthology.orchestra.run.smithi139.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-11-20T05:46:29.530 INFO:teuthology.orchestra.run.smithi139.stdout:customizable bootloader with modular architecture. It supports a rich 2021-11-20T05:46:29.530 INFO:teuthology.orchestra.run.smithi139.stdout:variety of kernel formats, file systems, computer architectures and 2021-11-20T05:46:29.530 INFO:teuthology.orchestra.run.smithi139.stdout:hardware devices. 2021-11-20T05:46:29.530 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:46:29.531 INFO:teuthology.orchestra.run.smithi139.stdout:This subpackage provides tools for support of all platforms. 2021-11-20T05:46:29.532 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-11-20T05:46:29.532 INFO:teuthology.task.kernel:Updating grub on smithi139 to boot 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:29.532 DEBUG:teuthology.orchestra.run.smithi139:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-11-20T05:46:30.909 INFO:teuthology.orchestra.run.smithi178.stderr:Generating grub configuration file ... 2021-11-20T05:46:31.128 INFO:teuthology.orchestra.run.smithi139.stderr:Generating grub configuration file ... 2021-11-20T05:46:33.175 INFO:teuthology.orchestra.run.smithi178.stderr:done 2021-11-20T05:46:33.177 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2021-11-20T05:46:33.198 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.uFv3mo0nqc 2021-11-20T05:46:33.199 DEBUG:teuthology.orchestra.run.smithi178:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uFv3mo0nqc 2021-11-20T05:46:33.269 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.uFv3mo0nqc 2021-11-20T05:46:33.403 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.uFv3mo0nqc is 6KB 2021-11-20T05:46:33.417 INFO:teuthology.orchestra.run.smithi139.stderr:done 2021-11-20T05:46:33.419 DEBUG:teuthology.orchestra.run.smithi139:> mktemp 2021-11-20T05:46:33.452 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.uFv3mo0nqc 2021-11-20T05:46:33.454 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.gJEPsCRyNS 2021-11-20T05:46:33.455 DEBUG:teuthology.orchestra.run.smithi139:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gJEPsCRyNS 2021-11-20T05:46:33.468 DEBUG:teuthology.orchestra.run.smithi178:> sudo /bin/ls /boot/loader/entries || true 2021-11-20T05:46:33.527 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.gJEPsCRyNS 2021-11-20T05:46:33.530 INFO:teuthology.orchestra.run.smithi178.stdout:3331cc438d4e47d3b31025a6d3bd39f2-0-rescue.conf 2021-11-20T05:46:33.530 INFO:teuthology.orchestra.run.smithi178.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-240.22.1.el8_3.x86_64.conf 2021-11-20T05:46:33.530 INFO:teuthology.orchestra.run.smithi178.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-301.1.el8.x86_64.conf 2021-11-20T05:46:33.530 INFO:teuthology.orchestra.run.smithi178.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64.conf 2021-11-20T05:46:33.531 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-set-default 3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:33.676 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.gJEPsCRyNS is 6KB 2021-11-20T05:46:33.692 DEBUG:teuthology.orchestra.run.smithi178:> sudo shutdown -r now 2021-11-20T05:46:33.724 DEBUG:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.gJEPsCRyNS 2021-11-20T05:46:33.739 DEBUG:teuthology.orchestra.run.smithi139:> sudo /bin/ls /boot/loader/entries || true 2021-11-20T05:46:33.812 INFO:teuthology.orchestra.run.smithi139.stdout:3331cc438d4e47d3b31025a6d3bd39f2-0-rescue.conf 2021-11-20T05:46:33.812 INFO:teuthology.orchestra.run.smithi139.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-240.22.1.el8_3.x86_64.conf 2021-11-20T05:46:33.812 INFO:teuthology.orchestra.run.smithi139.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-301.1.el8.x86_64.conf 2021-11-20T05:46:33.813 INFO:teuthology.orchestra.run.smithi139.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64.conf 2021-11-20T05:46:33.814 DEBUG:teuthology.orchestra.run.smithi139:> sudo grub2-set-default 3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:46:33.963 DEBUG:teuthology.orchestra.run.smithi139:> sudo shutdown -r now 2021-11-20T05:47:03.723 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2021-11-20T05:47:03.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2021-11-20T05:47:03.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:47:03.968 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2021-11-20T05:47:03.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2021-11-20T05:47:03.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:48:22.235 DEBUG:teuthology.orchestra.remote:timed out 2021-11-20T05:48:22.447 DEBUG:teuthology.orchestra.remote:timed out 2021-11-20T05:48:52.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2021-11-20T05:48:52.268 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:48:52.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2021-11-20T05:48:52.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:48:52.599 DEBUG:teuthology.orchestra.run.smithi178:> true 2021-11-20T05:48:52.820 DEBUG:teuthology.orchestra.run.smithi139:> true 2021-11-20T05:48:52.877 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2021-11-20T05:48:52.877 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-348.2.1.el8_5.x86_64"... 2021-11-20T05:48:52.878 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2021-11-20T05:48:52.894 INFO:teuthology.orchestra.run.smithi178.stdout:4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:48:52.895 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-348.2.1.el8_5.x86_64 vs 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:48:52.895 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-11-20T05:48:52.896 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-11-20T05:48:53.105 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2021-11-20T05:48:53.106 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-348.2.1.el8_5.x86_64"... 2021-11-20T05:48:53.106 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2021-11-20T05:48:53.123 INFO:teuthology.orchestra.run.smithi139.stdout:4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:48:53.123 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-348.2.1.el8_5.x86_64 vs 4.18.0-348.2.1.el8_5.x86_64 2021-11-20T05:48:53.124 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-11-20T05:48:53.124 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-11-20T05:48:53.897 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2021-11-20T05:48:53.897 DEBUG:teuthology.orchestra.run.smithi178:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-11-20T05:48:53.980 INFO:teuthology.orchestra.run.smithi178.stdout:ttyS1 2021-11-20T05:48:53.995 DEBUG:teuthology.parallel:result is None 2021-11-20T05:48:54.125 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2021-11-20T05:48:54.126 DEBUG:teuthology.orchestra.run.smithi139:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-11-20T05:48:54.222 INFO:teuthology.orchestra.run.smithi139.stdout:ttyS1 2021-11-20T05:48:54.235 DEBUG:teuthology.parallel:result is None 2021-11-20T05:48:54.237 INFO:teuthology.run_tasks:Running task internal.base... 2021-11-20T05:48:54.255 INFO:teuthology.task.internal:Creating test directory... 2021-11-20T05:48:54.255 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-11-20T05:48:54.258 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-11-20T05:48:54.277 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-11-20T05:48:54.287 INFO:teuthology.run_tasks:Running task internal.archive... 2021-11-20T05:48:54.306 INFO:teuthology.task.internal:Creating archive directory... 2021-11-20T05:48:54.307 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-11-20T05:48:54.316 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-11-20T05:48:54.346 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-11-20T05:48:54.356 INFO:teuthology.task.internal:Enabling coredump saving... 2021-11-20T05:48:54.356 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-11-20T05:48:54.383 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-11-20T05:48:54.409 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T05:48:54.417 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T05:48:54.420 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T05:48:54.429 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-20T05:48:54.430 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-11-20T05:48:54.440 INFO:teuthology.task.internal:Configuring sudo... 2021-11-20T05:48:54.441 DEBUG:teuthology.orchestra.run.smithi139:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-11-20T05:48:54.464 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-11-20T05:48:54.498 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-11-20T05:48:54.511 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-11-20T05:48:54.511 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-11-20T05:48:54.532 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-11-20T05:48:54.554 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T05:48:54.628 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T05:48:54.692 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T05:48:54.733 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T05:48:54.776 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-11-20T05:48:54.776 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-11-20T05:48:54.841 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T05:48:54.894 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-20T05:48:54.946 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T05:48:55.030 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-20T05:48:55.073 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2021-11-20T05:48:55.073 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-11-20T05:48:55.138 DEBUG:teuthology.orchestra.run.smithi139:> sudo service rsyslog restart 2021-11-20T05:48:55.141 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2021-11-20T05:48:55.190 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T05:48:55.233 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T05:48:55.709 INFO:teuthology.run_tasks:Running task internal.timer... 2021-11-20T05:48:55.722 INFO:teuthology.task.internal:Starting timer... 2021-11-20T05:48:55.723 INFO:teuthology.run_tasks:Running task pcp... 2021-11-20T05:48:55.737 INFO:teuthology.run_tasks:Running task selinux... 2021-11-20T05:48:55.754 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-11-20T05:48:55.754 DEBUG:teuthology.orchestra.run.smithi139:> sudo service auditd rotate 2021-11-20T05:48:55.808 INFO:teuthology.orchestra.run.smithi139.stdout:Rotating logs: [ OK ] 2021-11-20T05:48:55.809 DEBUG:teuthology.orchestra.run.smithi178:> sudo service auditd rotate 2021-11-20T05:48:55.867 INFO:teuthology.orchestra.run.smithi178.stdout:Rotating logs: [ OK ] 2021-11-20T05:48:55.869 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-11-20T05:48:55.869 DEBUG:teuthology.orchestra.run.smithi139:> /usr/sbin/getenforce 2021-11-20T05:48:55.893 INFO:teuthology.orchestra.run.smithi139.stdout:Permissive 2021-11-20T05:48:55.894 DEBUG:teuthology.orchestra.run.smithi178:> /usr/sbin/getenforce 2021-11-20T05:48:55.927 INFO:teuthology.orchestra.run.smithi178.stdout:Permissive 2021-11-20T05:48:55.928 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi139.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi178.front.sepia.ceph.com': 'permissive'} 2021-11-20T05:48:55.928 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T05:48:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T05:48:55.961 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T05:48:55.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T05:48:55.995 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-11-20T05:48:55.995 DEBUG:teuthology.orchestra.run.smithi139:> sudo /usr/sbin/setenforce permissive 2021-11-20T05:48:56.032 DEBUG:teuthology.orchestra.run.smithi178:> sudo /usr/sbin/setenforce permissive 2021-11-20T05:48:56.068 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-11-20T05:48:56.079 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-11-20T05:48:56.080 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-11-20T05:48:56.098 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-11-20T05:48:56.100 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi139.front.sepia.ceph.com,smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-11-20T05:48:56.463 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-11-20T05:48:57.318 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-11-20T05:49:00.241 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-11-20T05:49:00.350 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:00 +0000 (0:00:00.126) 0:00:00.126 ***** 2021-11-20T05:49:00.362 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:00 +0000 (0:00:00.011) 0:00:00.137 ***** 2021-11-20T05:49:01.737 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi178.front.sepia.ceph.com] 2021-11-20T05:49:01.788 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:01 +0000 (0:00:01.426) 0:00:01.563 ***** 2021-11-20T05:49:01.798 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] 2021-11-20T05:49:01.845 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:01 +0000 (0:00:00.057) 0:00:01.621 ***** 2021-11-20T05:49:02.055 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-11-20T05:49:02.057 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-20T05:49:02.079 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:02 +0000 (0:00:00.233) 0:00:01.855 ***** 2021-11-20T05:49:02.099 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:02 +0000 (0:00:00.019) 0:00:01.874 ***** 2021-11-20T05:49:02.595 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-11-20T05:49:02.597 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-11-20T05:49:02.617 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:02 +0000 (0:00:00.518) 0:00:02.393 ***** 2021-11-20T05:49:02.624 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-11-20T05:49:02.648 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:02 +0000 (0:00:00.030) 0:00:02.423 ***** 2021-11-20T05:49:02.675 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-11-20T05:49:02.677 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:49:02.698 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:02 +0000 (0:00:00.049) 0:00:02.473 ***** 2021-11-20T05:49:02.714 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:49:02.736 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:02 +0000 (0:00:00.038) 0:00:02.511 ***** 2021-11-20T05:49:02.962 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-11-20T05:49:02.964 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:49:02.985 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:02 +0000 (0:00:00.249) 0:00:02.761 ***** 2021-11-20T05:49:03.004 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:03 +0000 (0:00:00.018) 0:00:02.779 ***** 2021-11-20T05:49:03.828 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-11-20T05:49:03.830 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387343.032602-7910-244256194445769/source", "state": "file", "uid": 0} 2021-11-20T05:49:03.850 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:03 +0000 (0:00:00.846) 0:00:03.626 ***** 2021-11-20T05:49:03.857 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387343.0482476-7912-166041540388865/source", "state": "file", "uid": 0} 2021-11-20T05:49:03.880 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:03 +0000 (0:00:00.029) 0:00:03.655 ***** 2021-11-20T05:49:04.385 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-11-20T05:49:04.386 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-11-20T05:49:04.388 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-11-20T05:49:04.407 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-11-20T05:49:04.408 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-11-20T05:49:04.481 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-11-20T05:49:04.482 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-11-20T05:49:04.494 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-11-20T05:49:04.496 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:04 +0000 (0:00:00.614) 0:00:04.270 ***** 2021-11-20T05:49:05.006 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] 2021-11-20T05:49:05.028 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] 2021-11-20T05:49:05.049 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** 2021-11-20T05:49:05.051 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:05 +0000 (0:00:00.554) 0:00:04.825 ***** 2021-11-20T05:49:05.080 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-11-20T05:49:05.092 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-11-20T05:49:05.097 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-11-20T05:49:05.253 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:05 +0000 (0:00:00.203) 0:00:05.028 ***** 2021-11-20T05:49:05.271 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:05 +0000 (0:00:00.017) 0:00:05.046 ***** 2021-11-20T05:49:05.914 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi178.front.sepia.ceph.com] 2021-11-20T05:49:05.965 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-11-20T05:49:05.967 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:05 +0000 (0:00:00.694) 0:00:05.741 ***** 2021-11-20T05:49:06.621 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-11-20T05:49:06.623 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.166565", "end": "2021-11-20 05:49:06.599104", "rc": 0, "start": "2021-11-20 05:49:06.432539", "stderr": "", "stderr_lines": [], "stdout": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "stdout_lines": ["d72e06d3c78c8d9639a8ca75effa043ee468910f"]} 2021-11-20T05:49:06.643 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:06 +0000 (0:00:00.677) 0:00:06.418 ***** 2021-11-20T05:49:06.858 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.024351", "end": "2021-11-20 05:49:06.844026", "failed_when_result": false, "rc": 0, "start": "2021-11-20 05:49:06.819675", "stderr": "", "stderr_lines": [], "stdout": "2f6366fd31821994f3d217ad18c1675bf42a2115", "stdout_lines": ["2f6366fd31821994f3d217ad18c1675bf42a2115"]} 2021-11-20T05:49:06.883 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:06 +0000 (0:00:00.239) 0:00:06.658 ***** 2021-11-20T05:49:06.903 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi139.front.sepia.ceph.com] 2021-11-20T05:49:06.956 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:06 +0000 (0:00:00.073) 0:00:06.731 ***** 2021-11-20T05:49:06.962 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-11-20T05:49:06.964 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:49:06.986 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:06 +0000 (0:00:00.029) 0:00:06.761 ***** 2021-11-20T05:49:07.054 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-11-20T05:49:07.056 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:49:07.076 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:07 +0000 (0:00:00.089) 0:00:06.851 ***** 2021-11-20T05:49:07.145 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-11-20T05:49:07.146 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:49:07.171 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:07 +0000 (0:00:00.091) 0:00:06.943 ***** 2021-11-20T05:49:07.235 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-11-20T05:49:07.237 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:49:07.257 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:07 +0000 (0:00:00.089) 0:00:07.032 ***** 2021-11-20T05:49:07.337 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-11-20T05:49:07.339 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:49:07.359 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:07 +0000 (0:00:00.101) 0:00:07.134 ***** 2021-11-20T05:49:07.367 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-11-20T05:49:07.368 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.180389", "end": "2021-11-20 05:49:07.343141", "rc": 0, "start": "2021-11-20 05:49:07.162752", "stderr": "", "stderr_lines": [], "stdout": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "stdout_lines": ["d72e06d3c78c8d9639a8ca75effa043ee468910f"]} 2021-11-20T05:49:07.391 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:07 +0000 (0:00:00.032) 0:00:07.167 ***** 2021-11-20T05:49:07.589 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-11-20T05:49:07.591 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:49:07.592 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.004868", "end": "2021-11-20 05:49:07.582240", "failed_when_result": false, "rc": 0, "start": "2021-11-20 05:49:07.577372", "stderr": "", "stderr_lines": [], "stdout": "2f6366fd31821994f3d217ad18c1675bf42a2115", "stdout_lines": ["2f6366fd31821994f3d217ad18c1675bf42a2115"]} 2021-11-20T05:49:07.624 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:07 +0000 (0:00:00.232) 0:00:07.399 ***** 2021-11-20T05:49:07.646 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:07 +0000 (0:00:00.022) 0:00:07.422 ***** 2021-11-20T05:49:07.695 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-11-20T05:49:07.697 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:49:07.718 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:07 +0000 (0:00:00.071) 0:00:07.494 ***** 2021-11-20T05:49:07.799 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-11-20T05:49:07.801 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:49:07.822 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:07 +0000 (0:00:00.103) 0:00:07.597 ***** 2021-11-20T05:49:07.902 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-11-20T05:49:07.904 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:49:07.929 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:07 +0000 (0:00:00.106) 0:00:07.704 ***** 2021-11-20T05:49:07.939 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-11-20T05:49:07.942 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-11-20T05:49:08.004 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:49:08.026 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:08 +0000 (0:00:00.097) 0:00:07.801 ***** 2021-11-20T05:49:08.098 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-11-20T05:49:08.099 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:49:08.124 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:08 +0000 (0:00:00.098) 0:00:07.900 ***** 2021-11-20T05:49:08.141 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-11-20T05:49:08.143 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-11-20T05:49:08.321 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-11-20T05:49:08.364 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-11-20T05:49:08.365 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:49:08.386 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:49:08 +0000 (0:00:00.261) 0:00:08.162 ***** 2021-11-20T05:49:08.518 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-11-20T05:49:08.520 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-11-20T05:49:08.668 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-11-20T05:49:08.722 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-11-20T05:49:08.872 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-11-20T05:49:08.916 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-11-20T05:49:09.077 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-11-20T05:49:09.121 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-11-20T05:49:09.271 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-11-20T05:49:09.315 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-11-20T05:49:09.473 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-11-20T05:49:09.506 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-11-20T05:49:09.665 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-11-20T05:49:09.707 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-11-20T05:49:09.864 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-11-20T05:49:09.907 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-11-20T05:49:10.067 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-11-20T05:49:10.099 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-11-20T05:49:10.266 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-11-20T05:49:10.299 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-11-20T05:49:10.456 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-11-20T05:49:10.500 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-11-20T05:49:10.655 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-11-20T05:49:10.698 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-11-20T05:49:10.854 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-11-20T05:49:10.887 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-11-20T05:49:11.053 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-11-20T05:49:11.085 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-11-20T05:49:11.254 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-11-20T05:49:11.286 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-11-20T05:49:11.443 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-11-20T05:49:11.476 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-11-20T05:49:11.644 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-11-20T05:49:11.677 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-11-20T05:49:11.846 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-11-20T05:49:11.880 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-11-20T05:49:12.039 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-11-20T05:49:12.072 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-11-20T05:49:12.245 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-11-20T05:49:12.282 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-11-20T05:49:12.442 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-11-20T05:49:12.475 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-11-20T05:49:12.642 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-11-20T05:49:12.674 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-11-20T05:49:12.842 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-11-20T05:49:12.874 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-11-20T05:49:13.043 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-11-20T05:49:13.076 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-11-20T05:49:13.246 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-11-20T05:49:13.280 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-11-20T05:49:13.449 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-11-20T05:49:13.485 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-11-20T05:49:13.654 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-11-20T05:49:13.687 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-11-20T05:49:13.849 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-11-20T05:49:13.882 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-11-20T05:49:14.063 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-11-20T05:49:14.099 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-11-20T05:49:14.279 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-11-20T05:49:14.291 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-11-20T05:49:14.491 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-11-20T05:49:14.492 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-11-20T05:49:14.694 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-11-20T05:49:14.883 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-11-20T05:49:14.884 INFO:teuthology.task.ansible.out: 2021-11-20T05:49:14.895 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-11-20T05:49:15.085 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-11-20T05:49:15.277 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-11-20T05:49:15.288 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-11-20T05:49:15.480 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-20T05:49:15.492 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-11-20T05:49:15.681 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-11-20T05:49:15.683 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-11-20T05:49:15.872 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-11-20T05:49:15.873 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-11-20T05:49:16.074 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-11-20T05:49:16.265 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T05:49:16.276 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-11-20T05:49:16.464 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-11-20T05:49:16.475 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-11-20T05:49:16.665 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-11-20T05:49:16.666 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-11-20T05:49:16.857 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-11-20T05:49:16.872 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-11-20T05:49:17.062 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T05:49:17.064 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-11-20T05:49:17.254 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-11-20T05:49:17.267 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-11-20T05:49:17.457 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-11-20T05:49:17.468 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-11-20T05:49:17.658 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-11-20T05:49:17.680 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-11-20T05:49:17.860 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-11-20T05:49:17.871 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-11-20T05:49:18.054 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-11-20T05:49:18.066 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-11-20T05:49:18.069 INFO:teuthology.task.ansible.out: 2021-11-20T05:49:18.258 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-11-20T05:49:18.270 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-11-20T05:49:18.451 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-11-20T05:49:18.463 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-11-20T05:49:18.653 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-11-20T05:49:18.665 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-11-20T05:49:18.868 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-11-20T05:49:19.059 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-11-20T05:49:19.060 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-11-20T05:49:19.264 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-11-20T05:49:19.276 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-11-20T05:49:19.464 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-11-20T05:49:19.466 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-11-20T05:49:19.665 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-20T05:49:19.677 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-11-20T05:49:19.867 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-11-20T05:49:19.878 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-11-20T05:49:20.067 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-11-20T05:49:20.069 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-11-20T05:49:20.258 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-11-20T05:49:20.269 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-11-20T05:49:20.462 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-11-20T05:49:20.474 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-20T05:49:20.653 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-11-20T05:49:20.664 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-11-20T05:49:20.855 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-11-20T05:49:20.857 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-11-20T05:49:21.048 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-11-20T05:49:21.059 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-11-20T05:49:21.247 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-11-20T05:49:21.270 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-11-20T05:49:21.448 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-11-20T05:49:21.460 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-11-20T05:49:21.638 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-11-20T05:49:21.670 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-11-20T05:49:21.840 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-11-20T05:49:21.862 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-11-20T05:49:22.042 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-11-20T05:49:22.054 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-11-20T05:49:22.245 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-11-20T05:49:22.256 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-11-20T05:49:22.450 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-11-20T05:49:22.463 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-11-20T05:49:22.652 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-11-20T05:49:22.675 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-11-20T05:49:22.854 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-11-20T05:49:22.876 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-11-20T05:49:23.048 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-11-20T05:49:23.071 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-11-20T05:49:23.255 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-11-20T05:49:23.288 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-11-20T05:49:23.457 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-11-20T05:49:23.491 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-11-20T05:49:23.660 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-11-20T05:49:23.694 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-11-20T05:49:23.857 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-11-20T05:49:23.890 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-11-20T05:49:24.049 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-11-20T05:49:24.119 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-11-20T05:49:24.247 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-11-20T05:49:24.323 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-11-20T05:49:24.461 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-11-20T05:49:24.527 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-11-20T05:49:24.655 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-11-20T05:49:24.731 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-11-20T05:49:24.857 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-11-20T05:49:24.941 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-11-20T05:49:25.058 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-11-20T05:49:25.154 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-11-20T05:49:25.250 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-11-20T05:49:25.345 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-11-20T05:49:25.450 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-11-20T05:49:25.547 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-11-20T05:49:25.642 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-11-20T05:49:25.746 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-11-20T05:49:25.853 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-11-20T05:49:25.947 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-11-20T05:49:26.042 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-11-20T05:49:26.148 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-11-20T05:49:26.253 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-11-20T05:49:26.349 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-11-20T05:49:26.445 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-11-20T05:49:26.539 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-11-20T05:49:26.648 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-11-20T05:49:26.743 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-11-20T05:49:26.848 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T05:49:26.943 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-11-20T05:49:27.037 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-11-20T05:49:27.132 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-11-20T05:49:27.247 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-11-20T05:49:27.331 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-11-20T05:49:27.436 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-11-20T05:49:27.530 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-11-20T05:49:27.635 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-11-20T05:49:27.729 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-20T05:49:27.835 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-11-20T05:49:27.919 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-11-20T05:49:28.035 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-11-20T05:49:28.124 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-11-20T05:49:28.232 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-11-20T05:49:28.316 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-11-20T05:49:28.421 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-11-20T05:49:28.517 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-11-20T05:49:28.622 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-11-20T05:49:28.718 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-11-20T05:49:28.824 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-11-20T05:49:28.908 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-11-20T05:49:29.022 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-11-20T05:49:29.106 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-11-20T05:49:29.221 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-11-20T05:49:29.305 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-11-20T05:49:29.421 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-11-20T05:49:29.505 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-11-20T05:49:29.620 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-11-20T05:49:29.714 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-11-20T05:49:29.818 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-11-20T05:49:29.913 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-11-20T05:49:30.009 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-11-20T05:49:30.106 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-11-20T05:49:30.210 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-11-20T05:49:30.305 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-11-20T05:49:30.410 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-11-20T05:49:30.505 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-11-20T05:49:30.610 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-11-20T05:49:30.709 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-11-20T05:49:30.804 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-11-20T05:49:30.911 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-11-20T05:49:30.998 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-11-20T05:49:31.104 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-11-20T05:49:31.199 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-11-20T05:49:31.294 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-11-20T05:49:31.399 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-11-20T05:49:31.494 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-11-20T05:49:31.600 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-11-20T05:49:31.695 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-11-20T05:49:31.789 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-11-20T05:49:31.895 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-11-20T05:49:31.990 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-11-20T05:49:32.097 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-11-20T05:49:32.192 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-11-20T05:49:32.298 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-11-20T05:49:32.385 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-11-20T05:49:32.490 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-11-20T05:49:32.586 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-11-20T05:49:32.691 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-11-20T05:49:32.786 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-11-20T05:49:32.881 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-11-20T05:49:32.986 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-11-20T05:49:33.082 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-11-20T05:49:33.177 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-11-20T05:49:33.283 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-11-20T05:49:33.379 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-11-20T05:49:33.485 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-11-20T05:49:33.569 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-11-20T05:49:33.675 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-11-20T05:49:33.770 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-11-20T05:49:33.874 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-11-20T05:49:33.970 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-11-20T05:49:34.075 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-11-20T05:49:34.160 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-11-20T05:49:34.280 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-11-20T05:49:34.482 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-11-20T05:49:34.715 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-11-20T05:49:34.928 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-11-20T05:49:35.127 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-11-20T05:49:35.788 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-11-20T05:49:36.681 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-11-20T05:49:37.487 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-11-20T05:49:38.364 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-11-20T05:49:38.990 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-11-20T05:49:39.927 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-11-20T05:49:40.476 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-11-20T05:49:41.477 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-11-20T05:49:41.999 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-11-20T05:49:42.977 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-11-20T05:49:43.509 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-11-20T05:49:44.442 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-11-20T05:49:45.127 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-11-20T05:49:45.956 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-11-20T05:49:46.647 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-11-20T05:49:47.460 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-11-20T05:49:48.251 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-11-20T05:49:49.001 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-11-20T05:49:49.748 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-11-20T05:49:50.490 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-11-20T05:49:51.397 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-11-20T05:49:52.042 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-11-20T05:49:52.976 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-11-20T05:49:53.600 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-11-20T05:49:54.545 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-11-20T05:49:55.215 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-11-20T05:49:56.152 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-11-20T05:49:56.782 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-11-20T05:49:57.727 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-11-20T05:49:58.402 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-11-20T05:49:59.390 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-11-20T05:49:59.995 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-11-20T05:50:01.000 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-11-20T05:50:01.551 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-11-20T05:50:02.598 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-11-20T05:50:03.166 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-11-20T05:50:04.281 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-11-20T05:50:04.786 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-11-20T05:50:06.023 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-11-20T05:50:06.468 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-11-20T05:50:07.698 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10122, "groups": "sudo", "home": "/home/choffman", "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "name": "choffman", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10122} 2021-11-20T05:50:07.821 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:50:07 +0000 (0:00:59.434) 0:01:07.596 ***** 2021-11-20T05:50:07.890 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-11-20T05:50:07.912 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:50:07 +0000 (0:00:00.090) 0:01:07.687 ***** 2021-11-20T05:50:08.013 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-11-20T05:50:08.015 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@desktop 2021-11-20T05:50:08.017 INFO:teuthology.task.ansible.out: QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2021-11-20T05:50:08.019 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed20 2021-11-20T05:50:08.021 INFO:teuthology.task.ansible.out:82caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name" 2021-11-20T05:50:08.022 INFO:teuthology.task.ansible.out:: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915 2021-11-20T05:50:08.024 INFO:teuthology.task.ansible.out:d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZ 2021-11-20T05:50:08.026 INFO:teuthology.task.ansible.out:YM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a9 2021-11-20T05:50:08.026 INFO:teuthology.task.ansible.out:5579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 2021-11-20T05:50:08.028 INFO:teuthology.task.ansible.out: 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp 2021-11-20T05:50:08.030 INFO:teuthology.task.ansible.out:", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3 2021-11-20T05:50:08.032 INFO:teuthology.task.ansible.out:624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9Jj 2021-11-20T05:50:08.034 INFO:teuthology.task.ansible.out:enDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "sh 2021-11-20T05:50:08.035 INFO:teuthology.task.ansible.out:yamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a 2021-11-20T05:50:08.037 INFO:teuthology.task.ansible.out:28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5 2021-11-20T05:50:08.037 INFO:teuthology.task.ansible.out:299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": " 2021-11-20T05:50:08.039 INFO:teuthology.task.ansible.out:zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}]}, "changed": false} 2021-11-20T05:50:08.065 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:50:08 +0000 (0:00:00.153) 0:01:07.840 ***** 2021-11-20T05:50:08.083 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-11-20T05:50:08.084 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10122, "groups": "sudo", "home": "/home/choffman", "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "name": "choffman", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10122} 2021-11-20T05:50:08.215 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:50:08 +0000 (0:00:00.149) 0:01:07.990 ***** 2021-11-20T05:50:08.285 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-11-20T05:50:08.306 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:50:08 +0000 (0:00:00.091) 0:01:08.081 ***** 2021-11-20T05:50:08.544 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-11-20T05:50:08.548 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@desktop 2021-11-20T05:50:08.549 INFO:teuthology.task.ansible.out: QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2021-11-20T05:50:08.554 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed20 2021-11-20T05:50:08.556 INFO:teuthology.task.ansible.out:82caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name" 2021-11-20T05:50:08.558 INFO:teuthology.task.ansible.out:: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915 2021-11-20T05:50:08.560 INFO:teuthology.task.ansible.out:d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZ 2021-11-20T05:50:08.562 INFO:teuthology.task.ansible.out:YM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "ep 2021-11-20T05:50:08.562 INFO:teuthology.task.ansible.out:uertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 2021-11-20T05:50:08.564 INFO:teuthology.task.ansible.out: 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp 2021-11-20T05:50:08.566 INFO:teuthology.task.ansible.out:", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3 2021-11-20T05:50:08.568 INFO:teuthology.task.ansible.out:624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9Jj 2021-11-20T05:50:08.570 INFO:teuthology.task.ansible.out:enDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "sh 2021-11-20T05:50:08.572 INFO:teuthology.task.ansible.out:yamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a 2021-11-20T05:50:08.574 INFO:teuthology.task.ansible.out:28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": " 2021-11-20T05:50:08.576 INFO:teuthology.task.ansible.out:zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctG 2021-11-20T05:50:08.577 INFO:teuthology.task.ansible.out:GLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}]}, "changed": false} 2021-11-20T05:50:08.605 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:50:08 +0000 (0:00:00.297) 0:01:08.379 ***** 2021-11-20T05:50:08.968 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-11-20T05:50:08.970 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"after": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "attempts": 1, "before": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "changed": false, "remote_url_changed": false} 2021-11-20T05:50:08.993 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:50:08 +0000 (0:00:00.389) 0:01:08.768 ***** 2021-11-20T05:50:09.279 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"after": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "attempts": 1, "before": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "changed": false, "remote_url_changed": false} 2021-11-20T05:50:09.301 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:50:09 +0000 (0:00:00.308) 0:01:09.076 ***** 2021-11-20T05:50:09.485 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-20T05:50:09.487 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-11-20T05:50:09.685 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-11-20T05:50:09.748 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-11-20T05:50:09.906 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-11-20T05:50:09.981 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-11-20T05:50:10.130 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-11-20T05:50:10.133 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-11-20T05:50:10.184 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-11-20T05:50:10.331 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-11-20T05:50:10.396 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-11-20T05:50:10.397 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-11-20T05:50:10.533 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-11-20T05:50:10.536 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-11-20T05:50:10.614 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-11-20T05:50:10.761 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-11-20T05:50:10.835 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-11-20T05:50:10.838 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-11-20T05:50:10.986 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-11-20T05:50:11.040 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-11-20T05:50:11.201 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-11-20T05:50:11.212 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:11.265 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-11-20T05:50:11.432 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-11-20T05:50:11.434 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-11-20T05:50:11.464 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-11-20T05:50:11.475 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:11.643 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-11-20T05:50:11.645 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-11-20T05:50:11.654 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:11.665 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:11.687 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-11-20T05:50:11.689 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-11-20T05:50:11.896 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-11-20T05:50:11.898 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-11-20T05:50:11.907 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-11-20T05:50:11.908 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:11.920 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:12.130 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-11-20T05:50:12.131 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-11-20T05:50:12.133 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-11-20T05:50:12.333 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-11-20T05:50:12.335 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-11-20T05:50:12.344 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-11-20T05:50:12.544 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-11-20T05:50:12.546 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-11-20T05:50:12.763 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-11-20T05:50:12.764 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-11-20T05:50:12.964 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-11-20T05:50:12.975 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-11-20T05:50:12.977 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-11-20T05:50:13.174 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-11-20T05:50:13.176 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-11-20T05:50:13.185 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-11-20T05:50:13.187 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-11-20T05:50:13.385 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-11-20T05:50:13.397 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-11-20T05:50:13.399 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-11-20T05:50:13.605 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-11-20T05:50:13.608 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-11-20T05:50:13.824 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-11-20T05:50:13.825 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-11-20T05:50:13.827 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-11-20T05:50:14.036 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-11-20T05:50:14.037 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-11-20T05:50:14.039 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-11-20T05:50:14.240 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-11-20T05:50:14.241 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-11-20T05:50:14.449 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-11-20T05:50:14.461 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-11-20T05:50:14.662 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-11-20T05:50:14.673 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-11-20T05:50:14.883 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-11-20T05:50:14.885 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-11-20T05:50:15.083 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-11-20T05:50:15.085 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-11-20T05:50:15.094 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-11-20T05:50:15.283 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-11-20T05:50:15.305 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-11-20T05:50:15.307 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-11-20T05:50:15.504 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-11-20T05:50:15.504 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-11-20T05:50:15.704 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-11-20T05:50:15.715 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-11-20T05:50:15.916 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-11-20T05:50:15.937 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-11-20T05:50:16.126 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T05:50:16.148 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-11-20T05:50:16.339 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-11-20T05:50:16.362 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T05:50:16.548 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-11-20T05:50:16.597 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-11-20T05:50:16.765 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-11-20T05:50:16.767 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-11-20T05:50:16.808 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-11-20T05:50:16.976 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-11-20T05:50:16.978 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-11-20T05:50:17.031 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-11-20T05:50:17.033 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-11-20T05:50:17.190 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-11-20T05:50:17.233 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-11-20T05:50:17.235 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-11-20T05:50:17.414 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-11-20T05:50:17.443 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-11-20T05:50:17.610 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-11-20T05:50:17.654 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-11-20T05:50:17.832 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-11-20T05:50:17.835 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-20T05:50:17.855 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-11-20T05:50:18.045 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-11-20T05:50:18.046 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-11-20T05:50:18.048 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-11-20T05:50:18.066 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-11-20T05:50:18.068 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-20T05:50:18.255 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-11-20T05:50:18.277 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-11-20T05:50:18.279 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-11-20T05:50:18.281 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-11-20T05:50:18.454 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-11-20T05:50:18.487 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-11-20T05:50:18.666 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-11-20T05:50:18.687 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-11-20T05:50:18.875 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-11-20T05:50:18.897 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-11-20T05:50:19.085 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-11-20T05:50:19.107 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-11-20T05:50:19.288 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-11-20T05:50:19.290 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-11-20T05:50:19.320 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-11-20T05:50:19.507 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-11-20T05:50:19.530 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-11-20T05:50:19.532 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-11-20T05:50:19.707 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-11-20T05:50:19.739 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-11-20T05:50:19.927 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-11-20T05:50:19.929 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-11-20T05:50:19.948 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-11-20T05:50:20.138 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-11-20T05:50:20.148 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-11-20T05:50:20.150 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-11-20T05:50:20.348 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-11-20T05:50:20.359 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-11-20T05:50:20.557 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-11-20T05:50:20.559 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-11-20T05:50:20.569 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-11-20T05:50:20.763 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-11-20T05:50:20.774 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-11-20T05:50:20.775 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-11-20T05:50:20.971 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-11-20T05:50:20.982 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-11-20T05:50:21.180 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-11-20T05:50:21.192 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-11-20T05:50:21.390 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T05:50:21.401 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-11-20T05:50:21.601 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-11-20T05:50:21.603 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-20T05:50:21.811 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-11-20T05:50:21.813 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-11-20T05:50:22.022 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user" 2021-11-20T05:50:22.024 INFO:teuthology.task.ansible.out:: "pbs1108", "validate_certs": true} 2021-11-20T05:50:22.231 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-11-20T05:50:22.242 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-11-20T05:50:22.441 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-11-20T05:50:22.452 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-11-20T05:50:22.654 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-11-20T05:50:22.655 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-11-20T05:50:22.863 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-11-20T05:50:22.864 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-11-20T05:50:23.073 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-11-20T05:50:23.084 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-11-20T05:50:23.281 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-11-20T05:50:23.435 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-11-20T05:50:23.488 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-11-20T05:50:23.520 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-11-20T05:50:23.688 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-11-20T05:50:23.730 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-11-20T05:50:23.897 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-11-20T05:50:23.940 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-11-20T05:50:24.107 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-11-20T05:50:24.149 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-11-20T05:50:24.316 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-11-20T05:50:24.369 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-11-20T05:50:24.525 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-11-20T05:50:24.578 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-11-20T05:50:24.725 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-11-20T05:50:24.778 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-11-20T05:50:24.936 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-11-20T05:50:24.938 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-11-20T05:50:24.989 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-11-20T05:50:25.146 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-11-20T05:50:25.198 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-11-20T05:50:25.200 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-11-20T05:50:25.355 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-11-20T05:50:25.398 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-11-20T05:50:25.566 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-11-20T05:50:25.610 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-11-20T05:50:25.779 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-11-20T05:50:25.780 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-11-20T05:50:25.821 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-11-20T05:50:25.988 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-11-20T05:50:26.031 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-11-20T05:50:26.132 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-11-20T05:50:26.209 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-11-20T05:50:26.231 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-11-20T05:50:26.421 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-11-20T05:50:26.447 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-11-20T05:50:26.635 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-11-20T05:50:26.647 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-11-20T05:50:26.839 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-11-20T05:50:26.860 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-11-20T05:50:27.048 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-11-20T05:50:27.070 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-11-20T05:50:27.268 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-11-20T05:50:27.278 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-11-20T05:50:27.476 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-11-20T05:50:27.478 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-11-20T05:50:27.487 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-11-20T05:50:27.685 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-11-20T05:50:27.697 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-11-20T05:50:27.699 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-11-20T05:50:27.885 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-11-20T05:50:27.896 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-11-20T05:50:28.096 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-11-20T05:50:28.108 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-11-20T05:50:28.310 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-11-20T05:50:28.321 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-11-20T05:50:28.512 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-11-20T05:50:28.534 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-11-20T05:50:28.734 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-11-20T05:50:28.746 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-11-20T05:50:28.946 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-11-20T05:50:28.947 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-11-20T05:50:29.146 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-11-20T05:50:29.164 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-11-20T05:50:29.166 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-20T05:50:29.364 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWL 2021-11-20T05:50:29.366 INFO:teuthology.task.ansible.out:Fugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-11-20T05:50:29.576 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/Xa 2021-11-20T05:50:29.578 INFO:teuthology.task.ansible.out:gFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-11-20T05:50:29.787 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-11-20T05:50:29.788 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-11-20T05:50:29.987 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-11-20T05:50:29.989 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-11-20T05:50:30.197 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/w 2021-11-20T05:50:30.199 INFO:teuthology.task.ansible.out:QGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-11-20T05:50:30.396 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-11-20T05:50:30.407 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-11-20T05:50:30.606 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-11-20T05:50:30.618 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-11-20T05:50:30.821 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-11-20T05:50:30.833 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-11-20T05:50:31.034 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-11-20T05:50:31.045 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-11-20T05:50:31.247 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-11-20T05:50:31.249 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-11-20T05:50:31.453 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-11-20T05:50:31.485 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-11-20T05:50:31.670 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-11-20T05:50:31.692 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-11-20T05:50:31.872 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-11-20T05:50:31.905 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-11-20T05:50:32.084 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-11-20T05:50:32.107 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-11-20T05:50:32.297 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-11-20T05:50:32.319 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-11-20T05:50:32.507 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-11-20T05:50:32.529 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-11-20T05:50:32.725 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-11-20T05:50:32.737 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-11-20T05:50:32.936 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-11-20T05:50:32.947 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-11-20T05:50:32.949 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-11-20T05:50:33.148 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-11-20T05:50:33.151 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-11-20T05:50:33.159 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-11-20T05:50:33.359 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-11-20T05:50:33.361 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-11-20T05:50:33.561 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-11-20T05:50:33.572 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-11-20T05:50:33.774 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-11-20T05:50:33.775 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-11-20T05:50:33.987 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-11-20T05:50:33.998 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-11-20T05:50:34.191 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-11-20T05:50:34.202 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-11-20T05:50:34.405 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-11-20T05:50:34.417 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-11-20T05:50:34.630 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-11-20T05:50:34.632 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-11-20T05:50:34.834 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-11-20T05:50:34.836 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-11-20T05:50:35.055 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-11-20T05:50:35.067 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-11-20T05:50:35.258 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-11-20T05:50:35.269 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-11-20T05:50:35.471 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-11-20T05:50:35.482 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-11-20T05:50:35.682 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-11-20T05:50:35.693 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4m0LAcC7XiFiXFtYcCvZqX1CtyV9JU+VUEawZJpWrGTOr8c+vIexq7+WKg+eIUMot4+dtfwG8lByS1VofglIe5/quv17Xj7dO/hF2WkV8O7RYt7NhzmrKg54yB73pbRpvqzGceWWDvGRsWQGzbY7ARqOp4FPIDJvgXABCkak+hmjXyXpvXN418CBDbDmxHWwFb9QtB/MpQFiFrfl1EMDj8DaXZNWgo5J/qytic6LgiyPHhHfhSylk4cl9TGftNatobpMjqBT0O2/plk/qf+7pHPuvGSgihjijdpEPzVfctXE2OmsQAIZgkzXeCqts+s6ZYdUKq1FvpV5rMM8ZLqfCkDjEzFENJS68YYE2EjxAEgMzj7h/sWc4B5I5pxkAqVAQSGjYRu6rCCbFz8PXUdTyW+Vs1YeylnlBMiCa4xE4gwvW3HlWEnumtyURd1PRENrGJsJ/0uX6bpltbSg4sGACRYpLPJMm3OhqI3Jm/J/e2NLjfCw+svU2447xnfJOh68= pdhange@vaquita\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCoeyVaMjhcGrWQB+4Rq+SSIi8+OY/kx2s1vofiGt1gu3RNtFmG0Df1VITJJ32dbzaW7kTNxaRvhldNd/n7yEMssG+Wg+a5WKvSudlbVMmscbSXdFuVbcCW1jNrTGCYrxL7Sn5e7NQEQfAsCbfXwTcYBw33NdBVovC+gs2V3sZGvX3Y0akZONNuVg1mhu0L4m1/5IegJh005KeilwetHA2A5Dr10RG6oyIgT7t4z5gZXsrOiG3UQBXaekILYV9jxMWoFXGkVLc8TpL9SUAdxE3Owc0u1XiPqYYlEN+zDvE0554Zj8Oc2ZO6HRXCZXQFvzXeCD12YswcOfkIDmJRF7GT prashant@mauli", "key_options": null, "k 2021-11-20T05:50:35.695 INFO:teuthology.task.ansible.out:eyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-11-20T05:50:35.895 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-11-20T05:50:35.907 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4m0LAcC7XiFiXFtYcCvZqX1CtyV9JU+VUEawZJpWrGTOr8c+vIexq7+WKg+eIUMot4+dtfwG8lByS1VofglIe5/quv17Xj7dO/hF2WkV8O7RYt7NhzmrKg54yB73pbRpvqzGceWWDvGRsWQGzbY7ARqOp4FPIDJvgXABCkak+hmjXyXpvXN418CBDbDmxHWwFb9QtB/MpQFiFrfl1EMDj8DaXZNWgo5J/qytic6LgiyPHhHfhSylk4cl9TGftNatobpMjqBT0O2/plk/qf+7pHPuvGSgihjijdpEPzVfctXE2OmsQAIZgkzXeCqts+s6ZYdUKq1FvpV5rMM8ZLqfCkDjEzFENJS68YYE2EjxAEgMzj7h/sWc4B5I5pxkAqVAQSGjYRu6rCCbFz8PXUdTyW+Vs1YeylnlBMiCa4xE4gwvW3HlWEnumtyURd1PRENrGJsJ/0uX6bpltbSg4sGACRYpLPJMm3OhqI3Jm/J/e2NLjfCw+svU2447xnfJOh68= pdhange@vaquita\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCoeyVaMjhcGrWQB+4Rq+SSIi8+OY/kx2s1vofiGt1gu3RNtFmG0Df1VITJJ32dbzaW7kTNxaRvhldNd/n7yEMssG+Wg+a5WKvSudlbVMmscbSXdFuVbcCW1jNrTGCYrxL7Sn5e7NQEQfAsCbfXwTcYBw33NdBVovC+gs2V3sZGvX3Y0akZONNuVg1mhu0L4m1/5IegJh005KeilwetHA2A5Dr10RG6oyIgT7t4z5gZXsrOiG3UQBXaekILYV9jxMWoFXGkVLc8TpL9SUAdxE3Owc0u1XiPqYYlEN+zDvE0554Zj8Oc2ZO6HRXCZXQFvzXeCD12YswcOfkIDmJRF7GT prashant@mauli", "key_options": null, "k 2021-11-20T05:50:35.909 INFO:teuthology.task.ansible.out:eyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-11-20T05:50:36.121 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-11-20T05:50:36.132 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-11-20T05:50:36.346 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-11-20T05:50:36.348 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-11-20T05:50:36.549 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCt4sQTrDDg562IWY9Bw2fxFR7Y7b5lfo9egHa1qNFR5K4gt8lUwQXgzcxZK5dgSTZXHDGxrbYKWsdb8NbpgU/YjSqDUIptMZrhTI6pdHKZ2usY+nieaJj0NIZP/S7M93BbJeoNBYACjvDUr+vXEg6ZL3uAj+zFXr/4te+p4NqJBl4bSpvjcRK12O+F6HE3Q93j0Vlfs8IbXP4vcPcqOW+2usXw7JPdfjwuACJxrLTx/0mBsv8qXJDfOpPIHAMA/9SdTAzhs6XMo+fOwVOKkV+ki2fO3sl8pE16aFOkI3cYX36nkYYpZ2bJLnIPcLv37i3Nypl8eNLA4gzjkDf1o/KLO5vXOs76H189WGLqjh0SHAx/3KO6mrIhTq8Bv6YWoT3ntg22fXaxeajNhHcQ0bKZtGOAb0W1FcMRbSTkgEFAassYxlpTtuaCDIuA4fUVKW97Zu62eDDsSUtTWak/Yf940qAbptPPAw4ljmrvZRcjM9y6mGU8lKi5H4QoEOfbPok= nia@fedora", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-11-20T05:50:36.561 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-11-20T05:50:36.562 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-11-20T05:50:36.761 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-11-20T05:50:36.772 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCt4sQTrDDg562IWY9Bw2fxFR7Y7b5lfo9egHa1qNFR5K4gt8lUwQXgzcxZK5dgSTZXHDGxrbYKWsdb8NbpgU/YjSqDUIptMZrhTI6pdHKZ2usY+nieaJj0NIZP/S7M93BbJeoNBYACjvDUr+vXEg6ZL3uAj+zFXr/4te+p4NqJBl4bSpvjcRK12O+F6HE3Q93j0Vlfs8IbXP4vcPcqOW+2usXw7JPdfjwuACJxrLTx/0mBsv8qXJDfOpPIHAMA/9SdTAzhs6XMo+fOwVOKkV+ki2fO3sl8pE16aFOkI3cYX36nkYYpZ2bJLnIPcLv37i3Nypl8eNLA4gzjkDf1o/KLO5vXOs76H189WGLqjh0SHAx/3KO6mrIhTq8Bv6YWoT3ntg22fXaxeajNhHcQ0bKZtGOAb0W1FcMRbSTkgEFAassYxlpTtuaCDIuA4fUVKW97Zu62eDDsSUtTWak/Yf940qAbptPPAw4ljmrvZRcjM9y6mGU8lKi5H4QoEOfbPok= nia@fedora", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-11-20T05:50:36.981 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-11-20T05:50:36.983 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-11-20T05:50:37.181 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-11-20T05:50:37.192 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-11-20T05:50:37.394 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-11-20T05:50:37.407 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-11-20T05:50:37.606 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-11-20T05:50:37.617 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-11-20T05:50:37.815 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-11-20T05:50:37.827 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-11-20T05:50:38.027 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-11-20T05:50:38.029 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-11-20T05:50:38.229 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-11-20T05:50:38.240 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-11-20T05:50:38.438 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-11-20T05:50:38.450 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-11-20T05:50:38.649 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-11-20T05:50:38.661 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-11-20T05:50:38.866 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-11-20T05:50:38.868 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-11-20T05:50:39.069 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-11-20T05:50:39.080 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-11-20T05:50:39.279 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-11-20T05:50:39.291 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-11-20T05:50:39.491 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-11-20T05:50:39.503 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-11-20T05:50:39.702 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-11-20T05:50:39.714 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-11-20T05:50:39.907 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-11-20T05:50:39.918 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-11-20T05:50:40.130 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-11-20T05:50:40.138 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-11-20T05:50:40.348 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-11-20T05:50:40.360 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-11-20T05:50:40.560 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-11-20T05:50:40.592 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-11-20T05:50:40.775 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-11-20T05:50:40.808 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-11-20T05:50:40.988 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-11-20T05:50:41.022 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-11-20T05:50:41.191 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-11-20T05:50:41.234 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM7LPHiaPQXdE+yt9cn7UMuRr8k/LjSUmMrnrAqp8ySQtX6OA3/NcOsxg6FJ45chZqeoXwhsYEW+7pTInDqFlOlmVynR26lWnZfk9DjBJZcloUCn8y5MR9QitNYsOMs5KwCEFmBH6sP+2vCFocXBnKQcVZLRfip3MEG4LoWYfI4ir7fLmK1Hr5G+GhtMstXvVSG4VE4IgbU9/LRIOv7P9qFn4bRJoUSiS2dDbUxVIuneA9bIEnB9ZEVGahEzlUhx6qDSs0FvrnneCQ1ayOFpxClzjT5GuKk/J3LJvWAw8CAwDXCf3FzsWQsvb5ru5VSazUI9uA3o9wWkbLW2oZPgtzI8lmwbL+MT58T92tqaJOPID3IZDGjzNrRqU5BQYpvFYHpHRRiqV625og+ktYfXgO+dTbgAxScomqutydI4cyJusDtnWtljWnEocS+2hUSZpmB6B3Wj70TuuzsbYrcrJ/NgRpQNX4dwHCJl6vPT+xkoLh1imYAC7klf7cnrOyGcs= choffman@choffman.net", "key_options": null, "keyfile": "/home/choffman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "choffman", "validate_certs": true} 2021-11-20T05:50:41.324 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:50:41 +0000 (0:00:32.022) 0:01:41.099 ***** 2021-11-20T05:50:41.408 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM7LPHiaPQXdE+yt9cn7UMuRr8k/LjSUmMrnrAqp8ySQtX6OA3/NcOsxg6FJ45chZqeoXwhsYEW+7pTInDqFlOlmVynR26lWnZfk9DjBJZcloUCn8y5MR9QitNYsOMs5KwCEFmBH6sP+2vCFocXBnKQcVZLRfip3MEG4LoWYfI4ir7fLmK1Hr5G+GhtMstXvVSG4VE4IgbU9/LRIOv7P9qFn4bRJoUSiS2dDbUxVIuneA9bIEnB9ZEVGahEzlUhx6qDSs0FvrnneCQ1ayOFpxClzjT5GuKk/J3LJvWAw8CAwDXCf3FzsWQsvb5ru5VSazUI9uA3o9wWkbLW2oZPgtzI8lmwbL+MT58T92tqaJOPID3IZDGjzNrRqU5BQYpvFYHpHRRiqV625og+ktYfXgO+dTbgAxScomqutydI4cyJusDtnWtljWnEocS+2hUSZpmB6B3Wj70TuuzsbYrcrJ/NgRpQNX4dwHCJl6vPT+xkoLh1imYAC7klf7cnrOyGcs= choffman@choffman.net", "key_options": null, "keyfile": "/home/choffman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "choffman", "validate_certs": true} 2021-11-20T05:50:41.510 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:41.512 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd 2021-11-20T05:50:41.515 INFO:teuthology.task.ansible.out:0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible 2021-11-20T05:50:41.516 INFO:teuthology.task.ansible.out:_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:41.534 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:50:41 +0000 (0:00:00.210) 0:01:41.310 ***** 2021-11-20T05:50:41.616 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:41.627 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:41.638 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:41.640 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:41.649 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:41.660 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:41.671 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:41.673 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:41.683 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:41.726 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-11-20T05:50:41.739 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:41.749 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:41.907 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-11-20T05:50:41.918 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:41.920 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:41.951 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-11-20T05:50:42.152 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-11-20T05:50:42.163 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.166 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy V 2021-11-20T05:50:42.167 INFO:teuthology.task.ansible.out:Zrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.176 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.187 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.198 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.199 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.210 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.211 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.221 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.233 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.235 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.244 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.255 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.257 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.266 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.268 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.278 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.289 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.290 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.300 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.302 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.312 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.322 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.324 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.334 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.345 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.346 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.356 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.367 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.368 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.378 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-11-20T05:50:42.379 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.380 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.390 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.392 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.404 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.405 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.416 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.418 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.428 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.429 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.430 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.431 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.442 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.444 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.444 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.445 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False" 2021-11-20T05:50:42.447 INFO:teuthology.task.ansible.out:} 2021-11-20T05:50:42.456 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.458 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.468 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.470 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.480 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.481 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.492 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.503 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.505 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.515 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.516 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.526 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.528 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.538 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.540 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.551 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.552 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.562 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.564 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.574 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.576 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.586 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.588 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.598 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.600 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.610 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.611 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.622 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.623 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.633 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.635 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.636 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.646 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.648 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.658 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.660 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.671 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.672 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.682 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.684 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.684 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.695 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.697 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.697 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.708 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.710 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.720 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.722 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.732 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.732 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.733 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.733 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.744 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.745 INFO:teuthology.task.ansible.out: 2021-11-20T05:50:42.745 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.746 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.746 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.757 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.759 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.769 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.771 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.782 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.784 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.804 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.806 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.816 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.818 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.828 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.840 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.842 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.851 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.853 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.863 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.865 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.875 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.876 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.877 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.877 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.888 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.891 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.901 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.903 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.904 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.918 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.920 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.931 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.932 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.933 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.935 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.936 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.936 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.936 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db1571 2021-11-20T05:50:42.938 INFO:teuthology.task.ansible.out:4bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.947 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.949 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.960 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.963 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.973 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.975 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.986 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.988 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:42.998 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.000 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.010 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.012 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.024 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.035 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.037 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.037 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.037 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.048 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.050 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.061 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.062 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.072 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.074 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.085 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.086 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.097 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.099 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.099 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.110 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.112 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.121 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.123 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.134 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.136 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.146 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.150 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.158 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.160 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.171 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.173 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.184 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.187 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.196 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.208 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.230 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.231 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.232 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.255 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:50:43 +0000 (0:00:01.720) 0:01:43.030 ***** 2021-11-20T05:50:43.263 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.266 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.278 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.289 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.291 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.301 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.302 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.312 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.323 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} TASK [Filter the revoked_users list] ******************************************* 2021-11-20T05:50:43.325 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.348 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:50:43 +0000 (0:00:00.092) 0:01:43.123 ***** 2021-11-20T05:50:43.355 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.357 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.369 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.370 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.380 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.382 INFO:teuthology.task.ansible.out: skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.435 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:50:43 +0000 (0:00:00.087) 0:01:43.211 ***** 2021-11-20T05:50:43.505 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:50:43.527 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:50:43 +0000 (0:00:00.092) 0:01:43.303 ***** 2021-11-20T05:50:43.617 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-20T05:50:43.620 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-11-20T05:50:43.818 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-11-20T05:50:43.820 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-11-20T05:50:44.007 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-11-20T05:50:44.009 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-11-20T05:50:44.202 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-11-20T05:50:44.204 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-11-20T05:50:44.403 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-11-20T05:50:44.405 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-11-20T05:50:44.582 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-11-20T05:50:44.594 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-11-20T05:50:44.782 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-11-20T05:50:44.784 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-11-20T05:50:44.985 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-11-20T05:50:44.986 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-11-20T05:50:45.173 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-11-20T05:50:45.173 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-11-20T05:50:45.363 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-11-20T05:50:45.375 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-11-20T05:50:45.556 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-11-20T05:50:45.568 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-11-20T05:50:45.763 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} ok: [smithi139.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-11-20T05:50:45.951 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-11-20T05:50:46.219 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} ok: [smithi178.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} ok: [smithi139.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-11-20T05:50:46.358 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-11-20T05:50:46.391 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-11-20T05:50:46.547 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-11-20T05:50:46.590 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-11-20T05:50:46.738 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-11-20T05:50:46.778 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-11-20T05:50:46.939 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-11-20T05:50:46.981 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-11-20T05:50:47.127 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-11-20T05:50:47.181 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-11-20T05:50:47.319 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-11-20T05:50:47.363 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-11-20T05:50:47.524 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-11-20T05:50:47.556 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-11-20T05:50:47.722 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-11-20T05:50:47.765 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-11-20T05:50:47.912 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-11-20T05:50:47.954 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-11-20T05:50:48.100 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-11-20T05:50:48.142 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-11-20T05:50:48.300 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-11-20T05:50:48.342 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-11-20T05:50:48.489 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-11-20T05:50:48.533 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-11-20T05:50:48.682 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-11-20T05:50:48.724 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-11-20T05:50:48.881 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-11-20T05:50:48.923 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-11-20T05:50:49.069 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-11-20T05:50:49.111 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-11-20T05:50:49.267 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-11-20T05:50:49.310 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-11-20T05:50:49.459 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-11-20T05:50:49.502 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-11-20T05:50:49.648 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-11-20T05:50:49.701 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-11-20T05:50:49.847 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-11-20T05:50:49.890 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-11-20T05:50:50.035 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-11-20T05:50:50.089 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-11-20T05:50:50.224 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-11-20T05:50:50.288 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-11-20T05:50:50.427 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-11-20T05:50:50.490 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-11-20T05:50:50.625 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-11-20T05:50:50.679 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-11-20T05:50:50.814 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-11-20T05:50:50.877 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-11-20T05:50:51.002 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-11-20T05:50:51.075 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-11-20T05:50:51.267 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-11-20T05:50:52.568 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-11-20T05:50:52.824 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-11-20T05:50:54.152 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-11-20T05:50:54.354 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-11-20T05:50:55.774 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-11-20T05:50:55.900 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-11-20T05:50:57.410 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-11-20T05:50:57.431 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-11-20T05:50:58.910 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-11-20T05:50:59.046 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-11-20T05:51:00.425 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-11-20T05:51:00.615 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-11-20T05:51:00.641 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-11-20T05:51:00.819 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-11-20T05:51:00.841 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-11-20T05:51:01.011 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-11-20T05:51:01.054 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-11-20T05:51:01.212 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-11-20T05:51:01.243 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-11-20T05:51:01.400 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-11-20T05:51:01.443 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-11-20T05:51:01.602 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-11-20T05:51:01.634 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-11-20T05:51:01.801 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-11-20T05:51:01.823 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-11-20T05:51:01.993 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-11-20T05:51:02.016 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-11-20T05:51:02.196 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-11-20T05:51:02.218 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-11-20T05:51:02.403 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-11-20T05:51:02.426 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-11-20T05:51:02.461 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:02 +0000 (0:00:18.933) 0:02:02.236 ***** 2021-11-20T05:51:02.606 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-11-20T05:51:02.652 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:02 +0000 (0:00:00.190) 0:02:02.427 ***** 2021-11-20T05:51:03.041 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "checksum": "b3b89f3dabc35f78585fcbe6afcac330df9308f6", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "69c3df528743245bb36ff3b8fc484a02", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387462.5599422-19515-13260809022592/source", "state": "file", "uid": 0} 2021-11-20T05:51:03.066 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:03 +0000 (0:00:00.414) 0:02:02.841 ***** 2021-11-20T05:51:03.168 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-11-20T05:51:03.170 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => { "msg": "Host smithi139.front.sepia.ceph.com is running CentOS 8 (n/a)" } 2021-11-20T05:51:03.191 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:03 +0000 (0:00:00.125) 0:02:02.966 ***** 2021-11-20T05:51:03.242 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-11-20T05:51:03.243 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "checksum": "b3b89f3dabc35f78585fcbe6afcac330df9308f6", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "69c3df528743245bb36ff3b8fc484a02", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387462.744149-19528-268453460699939/source", "state": "file", "uid": 0} 2021-11-20T05:51:03.267 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:03 +0000 (0:00:00.075) 0:02:03.042 ***** 2021-11-20T05:51:03.286 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-11-20T05:51:03.287 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-11-20T05:51:03.311 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:03 +0000 (0:00:00.043) 0:02:03.086 ***** 2021-11-20T05:51:03.349 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-11-20T05:51:03.352 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => { "msg": "Host smithi178.front.sepia.ceph.com is running CentOS 8 (n/a)" } 2021-11-20T05:51:03.374 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:03 +0000 (0:00:00.062) 0:02:03.149 ***** 2021-11-20T05:51:03.392 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-11-20T05:51:03.393 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:03.416 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:03 +0000 (0:00:00.042) 0:02:03.191 ***** 2021-11-20T05:51:03.465 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-11-20T05:51:03.467 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-11-20T05:51:03.490 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:03 +0000 (0:00:00.074) 0:02:03.266 ***** 2021-11-20T05:51:03.591 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-11-20T05:51:03.593 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:03.615 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:03 +0000 (0:00:00.124) 0:02:03.390 ***** 2021-11-20T05:51:03.747 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-11-20T05:51:03.749 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.090231", "end": "2021-11-20 05:51:03.733560", "rc": 0, "start": "2021-11-20 05:51:03.643329", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-11-20T05:51:03.770 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:03 +0000 (0:00:00.155) 0:02:03.546 ***** 2021-11-20T05:51:03.852 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:03.877 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:03 +0000 (0:00:00.106) 0:02:03.652 ***** 2021-11-20T05:51:03.937 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-11-20T05:51:03.939 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.083338", "end": "2021-11-20 05:51:03.922232", "rc": 0, "start": "2021-11-20 05:51:03.838894", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-11-20T05:51:03.960 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:03 +0000 (0:00:00.083) 0:02:03.735 ***** 2021-11-20T05:51:03.967 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-11-20T05:51:03.969 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:03.993 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:03 +0000 (0:00:00.032) 0:02:03.768 ***** 2021-11-20T05:51:04.038 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:04.066 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.070) 0:02:03.838 ***** 2021-11-20T05:51:04.070 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:04.100 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.034) 0:02:03.872 ***** 2021-11-20T05:51:04.142 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-11-20T05:51:04.144 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:04.168 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.071) 0:02:03.944 ***** 2021-11-20T05:51:04.175 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-11-20T05:51:04.177 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:04.206 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.037) 0:02:03.981 ***** 2021-11-20T05:51:04.259 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:04.281 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.075) 0:02:04.056 ***** 2021-11-20T05:51:04.287 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-11-20T05:51:04.289 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:04.312 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.030) 0:02:04.087 ***** 2021-11-20T05:51:04.350 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-11-20T05:51:04.352 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:04.377 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.065) 0:02:04.153 ***** 2021-11-20T05:51:04.396 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-11-20T05:51:04.398 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:04.419 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.042) 0:02:04.195 ***** 2021-11-20T05:51:04.447 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-11-20T05:51:04.449 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:04.473 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.053) 0:02:04.249 ***** 2021-11-20T05:51:04.527 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi139.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:51:04.550 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.076) 0:02:04.326 ***** 2021-11-20T05:51:04.567 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-11-20T05:51:04.569 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:04.591 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.040) 0:02:04.367 ***** 2021-11-20T05:51:04.629 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:04.654 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.062) 0:02:04.429 ***** 2021-11-20T05:51:04.682 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-11-20T05:51:04.684 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi178.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:51:04.706 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.052) 0:02:04.481 ***** 2021-11-20T05:51:04.734 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-11-20T05:51:04.737 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:04.758 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.052) 0:02:04.534 ***** 2021-11-20T05:51:04.787 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-11-20T05:51:04.789 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:04.810 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.051) 0:02:04.585 ***** 2021-11-20T05:51:04.847 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-11-20T05:51:04.849 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:04.870 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.060) 0:02:04.646 ***** 2021-11-20T05:51:04.888 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-11-20T05:51:04.889 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:04.911 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.040) 0:02:04.686 ***** 2021-11-20T05:51:04.948 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-11-20T05:51:04.950 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:51:04.971 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:04 +0000 (0:00:00.060) 0:02:04.746 ***** 2021-11-20T05:51:04.996 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-11-20T05:51:04.998 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.019 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.048) 0:02:04.795 ***** 2021-11-20T05:51:05.056 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-11-20T05:51:05.058 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.082 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.062) 0:02:04.857 ***** 2021-11-20T05:51:05.099 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-11-20T05:51:05.100 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:51:05.121 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.039) 0:02:04.897 ***** 2021-11-20T05:51:05.148 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-11-20T05:51:05.150 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:51:05.171 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.049) 0:02:04.947 ***** 2021-11-20T05:51:05.188 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-11-20T05:51:05.190 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.212 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.040) 0:02:04.987 ***** 2021-11-20T05:51:05.238 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-11-20T05:51:05.240 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.262 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.050) 0:02:05.037 ***** 2021-11-20T05:51:05.279 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-11-20T05:51:05.281 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:51:05.302 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.040) 0:02:05.078 ***** 2021-11-20T05:51:05.329 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-11-20T05:51:05.331 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.353 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.050) 0:02:05.128 ***** 2021-11-20T05:51:05.370 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-11-20T05:51:05.372 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.393 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.040) 0:02:05.169 ***** 2021-11-20T05:51:05.420 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.444 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.050) 0:02:05.220 ***** 2021-11-20T05:51:05.461 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-11-20T05:51:05.463 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.485 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.040) 0:02:05.260 ***** 2021-11-20T05:51:05.512 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-11-20T05:51:05.514 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.536 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.051) 0:02:05.311 ***** 2021-11-20T05:51:05.554 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.578 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.042) 0:02:05.354 ***** 2021-11-20T05:51:05.627 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-11-20T05:51:05.629 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:51:05.651 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.072) 0:02:05.427 ***** 2021-11-20T05:51:05.658 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-11-20T05:51:05.660 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.683 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.031) 0:02:05.459 ***** 2021-11-20T05:51:05.722 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-11-20T05:51:05.725 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.746 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.062) 0:02:05.522 ***** 2021-11-20T05:51:05.764 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-11-20T05:51:05.765 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:51:05.788 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.041) 0:02:05.563 ***** 2021-11-20T05:51:05.816 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-11-20T05:51:05.818 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.840 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.052) 0:02:05.615 ***** 2021-11-20T05:51:05.858 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-11-20T05:51:05.860 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.886 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.046) 0:02:05.662 ***** 2021-11-20T05:51:05.914 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-11-20T05:51:05.916 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.941 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.054) 0:02:05.716 ***** 2021-11-20T05:51:05.962 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:05.985 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:05 +0000 (0:00:00.044) 0:02:05.761 ***** 2021-11-20T05:51:06.014 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-11-20T05:51:06.016 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:06.043 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:06 +0000 (0:00:00.058) 0:02:05.819 ***** 2021-11-20T05:51:06.064 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:06.087 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:06 +0000 (0:00:00.043) 0:02:05.863 ***** 2021-11-20T05:51:06.118 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-11-20T05:51:06.119 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:06.152 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:06 +0000 (0:00:00.064) 0:02:05.927 ***** 2021-11-20T05:51:06.172 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:06.195 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:06 +0000 (0:00:00.042) 0:02:05.970 ***** 2021-11-20T05:51:06.232 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-11-20T05:51:06.234 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:06.260 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:06 +0000 (0:00:00.065) 0:02:06.035 ***** 2021-11-20T05:51:06.277 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-11-20T05:51:06.279 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:06.301 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:06 +0000 (0:00:00.040) 0:02:06.076 ***** 2021-11-20T05:51:06.328 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-11-20T05:51:06.330 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:51:06.353 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:06 +0000 (0:00:00.051) 0:02:06.128 ***** 2021-11-20T05:51:06.371 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-11-20T05:51:06.373 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:06.396 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:06 +0000 (0:00:00.042) 0:02:06.171 ***** 2021-11-20T05:51:06.445 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-11-20T05:51:06.447 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:06.469 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:06 +0000 (0:00:00.073) 0:02:06.244 ***** 2021-11-20T05:51:06.487 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-11-20T05:51:06.489 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-20T05:51:06.511 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:06 +0000 (0:00:00.042) 0:02:06.287 ***** 2021-11-20T05:51:06.539 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-11-20T05:51:06.541 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:06.564 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:06 +0000 (0:00:00.052) 0:02:06.339 ***** 2021-11-20T05:51:06.593 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-11-20T05:51:06.595 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:06.617 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:06 +0000 (0:00:00.052) 0:02:06.392 ***** 2021-11-20T05:51:06.697 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-11-20T05:51:06.699 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:06.720 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:06 +0000 (0:00:00.103) 0:02:06.496 ***** 2021-11-20T05:51:06.845 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-11-20T05:51:06.846 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:51:06.870 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:06 +0000 (0:00:00.147) 0:02:06.644 ***** 2021-11-20T05:51:07.022 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:51:07.045 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:07 +0000 (0:00:00.177) 0:02:06.821 ***** 2021-11-20T05:51:07.712 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-11-20T05:51:07.715 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} ok: [smithi139.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-11-20T05:51:08.165 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-11-20T05:51:08.168 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-11-20T05:51:08.192 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:08 +0000 (0:00:01.146) 0:02:07.968 ***** 2021-11-20T05:51:08.212 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:08 +0000 (0:00:00.019) 0:02:07.987 ***** 2021-11-20T05:51:08.723 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-11-20T05:51:08.725 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-11-20T05:51:08.736 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-11-20T05:51:09.164 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-11-20T05:51:09.190 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.978) 0:02:08.966 ***** 2021-11-20T05:51:09.205 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-11-20T05:51:09.236 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.045) 0:02:09.011 ***** 2021-11-20T05:51:09.263 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-11-20T05:51:09.265 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.288 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.051) 0:02:09.063 ***** 2021-11-20T05:51:09.316 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.339 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.051) 0:02:09.115 ***** 2021-11-20T05:51:09.356 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-11-20T05:51:09.358 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.380 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.040) 0:02:09.155 ***** 2021-11-20T05:51:09.408 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.431 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.051) 0:02:09.207 ***** 2021-11-20T05:51:09.448 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-11-20T05:51:09.450 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.474 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.042) 0:02:09.249 ***** 2021-11-20T05:51:09.503 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.526 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.051) 0:02:09.301 ***** 2021-11-20T05:51:09.542 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-11-20T05:51:09.544 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.566 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.040) 0:02:09.342 ***** 2021-11-20T05:51:09.594 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.617 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.051) 0:02:09.393 ***** 2021-11-20T05:51:09.645 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-11-20T05:51:09.646 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.656 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.658 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.682 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.064) 0:02:09.457 ***** 2021-11-20T05:51:09.710 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.722 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.733 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.756 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.074) 0:02:09.532 ***** 2021-11-20T05:51:09.762 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-11-20T05:51:09.764 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.786 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.030) 0:02:09.562 ***** 2021-11-20T05:51:09.825 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.848 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.061) 0:02:09.623 ***** 2021-11-20T05:51:09.854 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-11-20T05:51:09.856 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.878 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.030) 0:02:09.654 ***** 2021-11-20T05:51:09.917 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.940 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.061) 0:02:09.716 ***** 2021-11-20T05:51:09.957 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-11-20T05:51:09.959 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:09.981 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:09 +0000 (0:00:00.041) 0:02:09.757 ***** 2021-11-20T05:51:10.010 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:10.033 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:10 +0000 (0:00:00.051) 0:02:09.808 ***** 2021-11-20T05:51:10.061 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-11-20T05:51:10.062 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:10.091 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:10 +0000 (0:00:00.057) 0:02:09.866 ***** 2021-11-20T05:51:10.112 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:10.149 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:10 +0000 (0:00:00.058) 0:02:09.925 ***** 2021-11-20T05:51:10.168 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-11-20T05:51:10.169 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:10.191 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:10 +0000 (0:00:00.041) 0:02:09.967 ***** 2021-11-20T05:51:10.218 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:10.242 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:10 +0000 (0:00:00.051) 0:02:10.018 ***** 2021-11-20T05:51:10.509 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-11-20T05:51:10.511 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.069963", "end": "2021-11-20 05:51:10.500332", "rc": 0, "start": "2021-11-20 05:51:10.430369", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.126876", "end": "2021-11-20 05:51:10.500319", "rc": 0, "start": "2021-11-20 05:51:10.373443", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:51:10.533 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:10 +0000 (0:00:00.290) 0:02:10.308 ***** 2021-11-20T05:51:10.556 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:10 +0000 (0:00:00.022) 0:02:10.331 ***** 2021-11-20T05:51:14.976 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:51:14.997 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:14 +0000 (0:00:04.440) 0:02:14.772 ***** 2021-11-20T05:51:15.079 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-11-20T05:51:15.080 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:15.104 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:15 +0000 (0:00:00.107) 0:02:14.879 ***** 2021-11-20T05:51:15.145 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-11-20T05:51:15.147 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:51:15.170 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:15 +0000 (0:00:00.065) 0:02:14.945 ***** 2021-11-20T05:51:15.187 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:15.210 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:15 +0000 (0:00:00.040) 0:02:14.986 ***** 2021-11-20T05:51:15.238 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-11-20T05:51:15.240 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:15.262 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:15 +0000 (0:00:00.051) 0:02:15.037 ***** 2021-11-20T05:51:15.279 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-11-20T05:51:15.281 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:15.303 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:15 +0000 (0:00:00.041) 0:02:15.078 ***** 2021-11-20T05:51:15.330 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-11-20T05:51:15.332 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:15.355 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:15 +0000 (0:00:00.051) 0:02:15.130 ***** 2021-11-20T05:51:15.372 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-11-20T05:51:15.374 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:15.396 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:15 +0000 (0:00:00.041) 0:02:15.172 ***** 2021-11-20T05:51:15.424 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-11-20T05:51:15.426 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:15.449 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:15 +0000 (0:00:00.052) 0:02:15.224 ***** 2021-11-20T05:51:15.518 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-11-20T05:51:15.519 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:15.540 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:15 +0000 (0:00:00.090) 0:02:15.315 ***** 2021-11-20T05:51:15.881 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-11-20T05:51:15.882 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-11-20T05:51:15.903 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:15 +0000 (0:00:00.363) 0:02:15.679 ***** 2021-11-20T05:51:16.037 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-11-20T05:51:16.060 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:16 +0000 (0:00:00.156) 0:02:15.836 ***** 2021-11-20T05:51:17.767 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:51:17.791 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:17 +0000 (0:00:01.729) 0:02:17.565 ***** 2021-11-20T05:51:17.922 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:51:17.945 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:17 +0000 (0:00:00.154) 0:02:17.720 ***** 2021-11-20T05:51:18.348 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-11-20T05:51:18.350 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-11-20T05:51:18.476 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-11-20T05:51:18.781 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-11-20T05:51:18.928 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-11-20T05:51:19.223 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-11-20T05:51:19.248 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:19 +0000 (0:00:01.302) 0:02:19.023 ***** 2021-11-20T05:51:19.350 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-11-20T05:51:19.378 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:19 +0000 (0:00:00.130) 0:02:19.154 ***** 2021-11-20T05:51:19.439 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-11-20T05:51:19.441 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:51:19.463 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:19 +0000 (0:00:00.085) 0:02:19.239 ***** 2021-11-20T05:51:19.574 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:51:19.598 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:19 +0000 (0:00:00.134) 0:02:19.373 ***** 2021-11-20T05:51:19.969 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-11-20T05:51:19.972 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-11-20T05:51:20.111 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-11-20T05:51:20.385 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-11-20T05:51:20.545 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-11-20T05:51:20.881 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387480.416571-20547-27566230279493/source", "state": "file", "uid": 0} 2021-11-20T05:51:20.939 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:20 +0000 (0:00:01.341) 0:02:20.715 ***** 2021-11-20T05:51:20.977 INFO:teuthology.task.ansible.out: TASK [common : Include dnf_system vars] **************************************** 2021-11-20T05:51:20.978 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/dnf_systems.yml"], "changed": false} 2021-11-20T05:51:21.001 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:20 +0000 (0:00:00.061) 0:02:20.776 ***** 2021-11-20T05:51:21.028 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-11-20T05:51:21.030 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387480.572513-20554-175802631533548/source", "state": "file", "uid": 0} 2021-11-20T05:51:21.065 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:21 +0000 (0:00:00.063) 0:02:20.840 ***** 2021-11-20T05:51:21.106 INFO:teuthology.task.ansible.out: TASK [common : Include dnf_system vars] **************************************** 2021-11-20T05:51:21.108 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/dnf_systems.yml"], "changed": false} 2021-11-20T05:51:21.129 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:21 +0000 (0:00:00.064) 0:02:20.904 ***** 2021-11-20T05:51:21.334 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* changed: [smithi178.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-11-20T05:51:21.355 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:21 +0000 (0:00:00.226) 0:02:21.131 ***** 2021-11-20T05:51:21.403 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-11-20T05:51:21.405 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T05:51:21.414 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T05:51:21.425 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T05:51:21.436 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T05:51:21.459 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:21 +0000 (0:00:00.103) 0:02:21.235 ***** 2021-11-20T05:51:22.200 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-11-20T05:51:22.202 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-11-20T05:51:22.223 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:22 +0000 (0:00:00.764) 0:02:21.999 ***** 2021-11-20T05:51:22.272 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-11-20T05:51:22.274 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T05:51:22.284 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T05:51:22.295 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-20T05:51:22.318 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:22 +0000 (0:00:00.094) 0:02:22.093 ***** 2021-11-20T05:51:23.300 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:51:23.323 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:23 +0000 (0:00:01.004) 0:02:23.098 ***** 2021-11-20T05:51:23.361 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (non-RHEL/CentOS)] ******** 2021-11-20T05:51:23.362 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:23.386 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:23 +0000 (0:00:00.062) 0:02:23.161 ***** 2021-11-20T05:51:23.859 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-11-20T05:51:23.861 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-11-20T05:51:23.882 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:23 +0000 (0:00:00.496) 0:02:23.658 ***** 2021-11-20T05:51:24.190 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-11-20T05:51:24.192 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:51:24.218 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:24 +0000 (0:00:00.335) 0:02:23.994 ***** 2021-11-20T05:51:24.247 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (non-RHEL/CentOS)] ******** 2021-11-20T05:51:24.249 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:24.269 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:24 +0000 (0:00:00.051) 0:02:24.045 ***** 2021-11-20T05:51:24.364 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-11-20T05:51:24.366 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-11-20T05:51:24.388 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:24 +0000 (0:00:00.118) 0:02:24.164 ***** 2021-11-20T05:51:24.426 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-11-20T05:51:24.428 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-11-20T05:51:24.450 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:24 +0000 (0:00:00.061) 0:02:24.225 ***** 2021-11-20T05:51:24.648 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-11-20T05:51:24.649 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:51:24.671 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:24 +0000 (0:00:00.220) 0:02:24.446 ***** 2021-11-20T05:51:24.741 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-11-20T05:51:24.743 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-11-20T05:51:24.766 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:24 +0000 (0:00:00.095) 0:02:24.541 ***** 2021-11-20T05:51:24.899 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-11-20T05:51:24.901 INFO:teuthology.task.ansible.out:fatal: [smithi178.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-11-20T05:51:24.923 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:24 +0000 (0:00:00.157) 0:02:24.699 ***** 2021-11-20T05:51:24.961 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-11-20T05:51:24.963 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:24.984 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:24 +0000 (0:00:00.061) 0:02:24.760 ***** 2021-11-20T05:51:25.239 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-11-20T05:51:25.240 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-11-20T05:51:25.268 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:25 +0000 (0:00:00.283) 0:02:25.043 ***** 2021-11-20T05:51:25.320 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-11-20T05:51:25.322 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-11-20T05:51:25.352 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:25 +0000 (0:00:00.083) 0:02:25.127 ***** 2021-11-20T05:51:25.453 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-11-20T05:51:25.455 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-11-20T05:51:25.479 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:25 +0000 (0:00:00.127) 0:02:25.254 ***** 2021-11-20T05:51:25.549 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-11-20T05:51:25.552 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:51:25.575 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:25 +0000 (0:00:00.096) 0:02:25.351 ***** 2021-11-20T05:51:25.801 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-11-20T05:51:25.803 INFO:teuthology.task.ansible.out:fatal: [smithi139.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-11-20T05:51:25.824 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:25 +0000 (0:00:00.248) 0:02:25.599 ***** 2021-11-20T05:51:25.851 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-11-20T05:51:25.853 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:51:25.874 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:25 +0000 (0:00:00.050) 0:02:25.649 ***** 2021-11-20T05:51:26.099 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-11-20T05:51:26.100 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 05:48:28 UTC", "ActiveEnterTimestampMonotonic": "19991549", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target sysinit.target tmp.mount systemd-tmpfiles-setup.service network-online.target var-run.mount basic.target -.mount systemd-journald.socket time-sync.target system.slice local-fs.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 05:48:28 UTC", "AssertTimestampMonotonic": "19983642", "Before": "plymouth-quit.service xdm.service shutdown.target multi-user.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 05:48:28 UTC", "Co 2021-11-20T05:51:26.102 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "19983642", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1274", "ExecMainStartTimestamp": "Sat 2021-11-20 05:48:28 UTC", "ExecMainStartTimestampMonotonic": "19991501", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Sat 2021-11-20 05:48:28 UTC] ; stop_time=[n/a] ; pid=1274 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 05:48:28 UTC", "InactiveExitTimestampMonotonic": "199915 2021-11-20T05:51:26.103 INFO:teuthology.task.ansible.out:49", "InvocationID": "47306c85d8d94e7484623ba7b09aa58a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1274", "MemoryAccounting": "yes", "MemoryCurrent": "667648", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectC 2021-11-20T05:51:26.105 INFO:teuthology.task.ansible.out:ontrolGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 05:48:28 UTC", "StateChangeTimestampMonotonic": "19991549", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203587", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 05:48:28 UTC", "WatchdogTimestampMonotonic": "19991547", "WatchdogUSec": "0"}} 2021-11-20T05:51:26.124 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:26 +0000 (0:00:00.250) 0:02:25.899 ***** 2021-11-20T05:51:26.320 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001745", "end": "2021-11-20 05:51:26.304989", "rc": 0, "start": "2021-11-20 05:51:26.303244", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-11-20T05:51:26.343 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:26 +0000 (0:00:00.219) 0:02:26.119 ***** 2021-11-20T05:51:26.351 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-11-20T05:51:26.352 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-11-20T05:51:26.376 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:26 +0000 (0:00:00.032) 0:02:26.152 ***** 2021-11-20T05:51:26.674 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-11-20T05:51:26.675 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 05:48:14 UTC", "ActiveEnterTimestampMonotonic": "20165690", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network-online.target var-run.mount tmp.mount sysinit.target basic.target nss-lookup.target systemd-tmpfiles-setup.service systemd-journald.socket local-fs.target -.mount system.slice time-sync.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 05:48:14 UTC", "AssertTimestampMonotonic": "20148833", "Before": "plymouth-quit.service xdm.service getty@tty1.service multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 05:48:14 UTC", "Co 2021-11-20T05:51:26.677 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "20148833", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1273", "ExecMainStartTimestamp": "Sat 2021-11-20 05:48:14 UTC", "ExecMainStartTimestampMonotonic": "20165624", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Sat 2021-11-20 05:48:14 UTC] ; stop_time=[n/a] ; pid=1273 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 05:48:14 UTC", "InactiveExitTimestampMonotonic": "201656 2021-11-20T05:51:26.678 INFO:teuthology.task.ansible.out:90", "InvocationID": "49e8ab5c3adb42e5b6799eeb6cf9639f", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1273", "MemoryAccounting": "yes", "MemoryCurrent": "679936", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectC 2021-11-20T05:51:26.680 INFO:teuthology.task.ansible.out:ontrolGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 05:48:14 UTC", "StateChangeTimestampMonotonic": "20165690", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 05:48:14 UTC", "WatchdogTimestampMonotonic": "20165684", "WatchdogUSec": "0"}} 2021-11-20T05:51:26.680 INFO:teuthology.task.ansible.out: 2021-11-20T05:51:26.704 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:26 +0000 (0:00:00.327) 0:02:26.479 ***** 2021-11-20T05:51:26.901 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-11-20T05:51:26.903 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001805", "end": "2021-11-20 05:51:26.885980", "rc": 0, "start": "2021-11-20 05:51:26.884175", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-11-20T05:51:26.924 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:26 +0000 (0:00:00.219) 0:02:26.698 ***** 2021-11-20T05:51:28.128 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-11-20T05:51:28.130 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:51:28.161 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:28 +0000 (0:00:01.230) 0:02:27.929 ***** 2021-11-20T05:51:28.732 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:51:28.756 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:51:28 +0000 (0:00:00.602) 0:02:28.531 ***** 2021-11-20T05:52:07.118 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-83.el8.noarch", "Installed: selinux-policy-targeted-3.14.3-83.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-11-20T05:52:07.143 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:07 +0000 (0:00:38.387) 0:03:06.919 ***** 2021-11-20T05:52:07.580 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-83.el8.noarch", "Installed: selinux-policy-targeted-3.14.3-83.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-11-20T05:52:07.602 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:07 +0000 (0:00:00.458) 0:03:07.377 ***** 2021-11-20T05:52:07.853 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-11-20T05:52:07.855 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-11-20T05:52:07.877 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:07 +0000 (0:00:00.275) 0:03:07.652 ***** 2021-11-20T05:52:08.064 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-11-20T05:52:08.087 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:08 +0000 (0:00:00.210) 0:03:07.863 ***** 2021-11-20T05:52:23.963 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-11-20T05:52:23.965 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.673500", "end": "2021-11-20 05:52:23.949316", "failed_when_result": false, "rc": 0, "start": "2021-11-20 05:52:08.275816", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-11-20T05:52:23.985 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:23 +0000 (0:00:15.897) 0:03:23.761 ***** 2021-11-20T05:52:24.452 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-11-20T05:52:24.454 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-11-20T05:52:24.475 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:24 +0000 (0:00:00.489) 0:03:24.251 ***** 2021-11-20T05:52:24.483 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-11-20T05:52:24.485 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.394626", "end": "2021-11-20 05:52:24.465791", "failed_when_result": false, "rc": 0, "start": "2021-11-20 05:52:08.071165", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-11-20T05:52:24.508 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:24 +0000 (0:00:00.032) 0:03:24.283 ***** 2021-11-20T05:52:24.682 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-11-20T05:52:24.684 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.009541", "end": "2021-11-20 05:52:24.668248", "rc": 0, "start": "2021-11-20 05:52:24.658707", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:52:24.705 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:24 +0000 (0:00:00.197) 0:03:24.481 ***** 2021-11-20T05:52:24.912 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-11-20T05:52:24.915 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.015584", "end": "2021-11-20 05:52:24.901059", "rc": 0, "start": "2021-11-20 05:52:24.885475", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:52:24.935 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:24 +0000 (0:00:00.229) 0:03:24.711 ***** 2021-11-20T05:52:24.974 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-11-20T05:52:24.976 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-11-20T05:52:25.001 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:24 +0000 (0:00:00.065) 0:03:24.777 ***** 2021-11-20T05:52:25.207 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-11-20T05:52:25.209 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.014116", "end": "2021-11-20 05:52:25.194233", "rc": 0, "start": "2021-11-20 05:52:25.180117", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:52:25.232 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:25 +0000 (0:00:00.231) 0:03:25.008 ***** 2021-11-20T05:52:25.450 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-11-20T05:52:25.452 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.029552", "end": "2021-11-20 05:52:25.442998", "rc": 0, "start": "2021-11-20 05:52:25.413446", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:52:25.483 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:25 +0000 (0:00:00.251) 0:03:25.259 ***** 2021-11-20T05:52:41.004 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-11-20T05:52:41.006 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.876585", "end": "2021-11-20 05:52:40.993696", "rc": 0, "start": "2021-11-20 05:52:25.117111", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:52:41.028 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:41 +0000 (0:00:15.544) 0:03:40.803 ***** 2021-11-20T05:52:41.233 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-11-20T05:52:41.235 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-11-20T05:52:41.256 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:41 +0000 (0:00:00.228) 0:03:41.031 ***** 2021-11-20T05:52:41.284 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-11-20T05:52:41.285 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-11-20T05:52:41.307 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:41 +0000 (0:00:00.050) 0:03:41.082 ***** 2021-11-20T05:52:41.314 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-11-20T05:52:41.316 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.615594", "end": "2021-11-20 05:52:41.283822", "rc": 0, "start": "2021-11-20 05:52:25.668228", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:52:41.339 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:41 +0000 (0:00:00.032) 0:03:41.114 ***** 2021-11-20T05:52:41.346 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-11-20T05:52:41.348 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-11-20T05:52:41.372 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:41 +0000 (0:00:00.032) 0:03:41.147 ***** 2021-11-20T05:52:41.411 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "Ce 2021-11-20T05:52:41.412 INFO:teuthology.task.ansible.out:ntOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T05:52:41.435 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:41 +0000 (0:00:00.063) 0:03:41.211 ***** 2021-11-20T05:52:41.474 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-11-20T05:52:41.476 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-20T05:52:41.478 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T05:52:41.499 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:41 +0000 (0:00:00.063) 0:03:41.274 ***** 2021-11-20T05:52:41.537 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-11-20T05:52:41.538 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-11-20T05:52:41.561 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:41 +0000 (0:00:00.061) 0:03:41.336 ***** 2021-11-20T05:52:41.588 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-11-20T05:52:41.589 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-11-20T05:52:41.610 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:41 +0000 (0:00:00.049) 0:03:41.386 ***** 2021-11-20T05:52:41.649 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* ok: [smithi178.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-11-20T05:52:41.672 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:41 +0000 (0:00:00.061) 0:03:41.448 ***** 2021-11-20T05:52:41.690 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-20T05:52:41.716 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:41 +0000 (0:00:00.042) 0:03:41.490 ***** 2021-11-20T05:52:41.721 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-11-20T05:52:41.722 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-20T05:52:41.723 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T05:52:41.753 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:41 +0000 (0:00:00.038) 0:03:41.529 ***** 2021-11-20T05:52:41.791 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-11-20T05:52:41.792 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-20T05:52:41.794 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-20T05:52:41.814 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:41 +0000 (0:00:00.060) 0:03:41.589 ***** 2021-11-20T05:52:41.914 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-11-20T05:52:41.937 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:41 +0000 (0:00:00.122) 0:03:41.712 ***** 2021-11-20T05:52:41.965 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-11-20T05:52:41.967 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:52:41.987 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:41 +0000 (0:00:00.050) 0:03:41.763 ***** 2021-11-20T05:52:42.005 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-11-20T05:52:42.007 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-20T05:52:42.029 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:42 +0000 (0:00:00.040) 0:03:41.803 ***** 2021-11-20T05:52:42.035 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-11-20T05:52:42.036 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:52:42.065 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:42 +0000 (0:00:00.037) 0:03:41.841 ***** 2021-11-20T05:52:42.217 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-11-20T05:52:42.219 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-11-20T05:52:42.241 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:42 +0000 (0:00:00.175) 0:03:42.016 ***** 2021-11-20T05:52:42.267 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-11-20T05:52:42.269 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:52:42.290 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:42 +0000 (0:00:00.048) 0:03:42.065 ***** 2021-11-20T05:52:42.295 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-11-20T05:52:42.297 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-11-20T05:52:42.321 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:42 +0000 (0:00:00.028) 0:03:42.094 ***** 2021-11-20T05:52:42.325 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-11-20T05:52:42.330 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:52:42.349 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:42 +0000 (0:00:00.030) 0:03:42.124 ***** 2021-11-20T05:52:42.562 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-11-20T05:52:42.564 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-11-20T05:52:42.584 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:42 +0000 (0:00:00.235) 0:03:42.360 ***** 2021-11-20T05:52:42.851 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-11-20T05:52:42.853 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-11-20T05:52:42.873 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:42 +0000 (0:00:00.289) 0:03:42.649 ***** 2021-11-20T05:52:42.900 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-11-20T05:52:42.902 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi139"}, "changed": false} 2021-11-20T05:52:42.923 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:42 +0000 (0:00:00.048) 0:03:42.698 ***** 2021-11-20T05:52:43.116 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-11-20T05:52:43.118 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-11-20T05:52:43.139 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:43 +0000 (0:00:00.216) 0:03:42.914 ***** 2021-11-20T05:52:43.176 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi178"}, "changed": false} 2021-11-20T05:52:43.197 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:43 +0000 (0:00:00.058) 0:03:42.973 ***** 2021-11-20T05:52:43.708 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-11-20T05:52:43.710 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi178.front.sepia.ceph.com", "ansible_hostname": "smithi178", "ansible_nodename": "smithi178"}, "changed": true, "name": "smithi178"} changed: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi139.front.sepia.ceph.com", "ansible_hostname": "smithi139", "ansible_nodename": "smithi139"}, "changed": true, "name": "smithi139"} 2021-11-20T05:52:43.730 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:43 +0000 (0:00:00.533) 0:03:43.506 ***** 2021-11-20T05:52:43.753 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:43 +0000 (0:00:00.021) 0:03:43.528 ***** 2021-11-20T05:52:44.200 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-11-20T05:52:44.202 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-11-20T05:52:44.222 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:44 +0000 (0:00:00.469) 0:03:43.998 ***** 2021-11-20T05:52:44.230 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-11-20T05:52:44.254 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:44 +0000 (0:00:00.031) 0:03:44.029 ***** 2021-11-20T05:52:44.712 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-11-20T05:52:44.714 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-11-20T05:52:44.736 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:44 +0000 (0:00:00.481) 0:03:44.511 ***** 2021-11-20T05:52:44.774 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-11-20T05:52:44.800 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:44 +0000 (0:00:00.064) 0:03:44.576 ***** 2021-11-20T05:52:45.270 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-11-20T05:52:45.272 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-11-20T05:52:45.294 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:45 +0000 (0:00:00.493) 0:03:45.069 ***** 2021-11-20T05:52:45.323 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-11-20T05:52:45.347 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:45 +0000 (0:00:00.053) 0:03:45.123 ***** 2021-11-20T05:52:45.765 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-11-20T05:52:45.768 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-11-20T05:52:45.798 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:45 +0000 (0:00:00.450) 0:03:45.573 ***** 2021-11-20T05:52:45.820 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-11-20T05:52:45.852 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:45 +0000 (0:00:00.053) 0:03:45.627 ***** 2021-11-20T05:52:46.008 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-20T05:52:46.010 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T05:52:46.042 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:46 +0000 (0:00:00.190) 0:03:45.818 ***** 2021-11-20T05:52:46.065 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T05:52:46.096 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:46 +0000 (0:00:00.053) 0:03:45.871 ***** 2021-11-20T05:52:46.248 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-11-20T05:52:46.250 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-11-20T05:52:46.275 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:46 +0000 (0:00:00.178) 0:03:46.050 ***** 2021-11-20T05:52:46.293 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-11-20T05:52:46.318 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:46 +0000 (0:00:00.041) 0:03:46.092 ***** 2021-11-20T05:52:46.491 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* ok: [smithi139.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-11-20T05:52:46.514 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:46 +0000 (0:00:00.197) 0:03:46.289 ***** 2021-11-20T05:52:46.542 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-11-20T05:52:46.566 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:46 +0000 (0:00:00.051) 0:03:46.342 ***** 2021-11-20T05:52:46.982 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-11-20T05:52:46.984 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-11-20T05:52:47.005 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:46 +0000 (0:00:00.439) 0:03:46.781 ***** 2021-11-20T05:52:47.007 INFO:teuthology.task.ansible.out: 2021-11-20T05:52:47.045 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-11-20T05:52:47.074 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:47 +0000 (0:00:00.068) 0:03:46.850 ***** 2021-11-20T05:52:47.480 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-11-20T05:52:47.483 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-11-20T05:52:47.503 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:47 +0000 (0:00:00.428) 0:03:47.278 ***** 2021-11-20T05:52:47.545 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-11-20T05:52:47.570 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:47 +0000 (0:00:00.067) 0:03:47.346 ***** 2021-11-20T05:52:50.102 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-20T05:52:50.103 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.401625", "end": "2021-11-20 05:52:50.089811", "rc": 0, "start": "2021-11-20 05:52:47.688186", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:52:50.125 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:50 +0000 (0:00:02.555) 0:03:49.901 ***** 2021-11-20T05:52:50.330 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.562649", "end": "2021-11-20 05:52:50.313051", "rc": 0, "start": "2021-11-20 05:52:47.750402", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:52:50.332 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ [WARNING]: Consider using the file module with state=absent rather than running 'rm'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rvf /etc/yum.repos.d/*-Stream-*", "delta": "0:00:00.002608", "end": "2021-11-20 05:52:50.325624", "rc": 0, "start": "2021-11-20 05:52:50.323016", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:52:50.353 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:50 +0000 (0:00:00.227) 0:03:50.128 ***** 2021-11-20T05:52:50.373 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:50 +0000 (0:00:00.019) 0:03:50.148 ***** 2021-11-20T05:52:50.390 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:52:50.391 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"dash_stream": "-stream"}, "changed": false} 2021-11-20T05:52:50.413 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:50 +0000 (0:00:00.040) 0:03:50.188 ***** 2021-11-20T05:52:50.460 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-11-20T05:52:50.462 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml) => {"ansible_facts": {"packages_to_upgrade": ["systemd"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"} 2021-11-20T05:52:50.483 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:50 +0000 (0:00:00.070) 0:03:50.259 ***** 2021-11-20T05:52:50.582 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-11-20T05:52:50.584 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rvf /etc/yum.repos.d/*-Stream-*", "delta": "0:00:00.002715", "end": "2021-11-20 05:52:50.574536", "rc": 0, "start": "2021-11-20 05:52:50.571821", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:52:50.606 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:50 +0000 (0:00:00.122) 0:03:50.382 ***** 2021-11-20T05:52:50.655 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:52:50.657 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"dash_stream": "-stream"}, "changed": false} 2021-11-20T05:52:50.677 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:50 +0000 (0:00:00.070) 0:03:50.452 ***** 2021-11-20T05:52:50.735 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-11-20T05:52:50.737 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml) => {"ansible_facts": {"packages_to_upgrade": ["systemd"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"} 2021-11-20T05:52:50.758 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:50 +0000 (0:00:00.081) 0:03:50.534 ***** 2021-11-20T05:52:50.962 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-11-20T05:52:50.964 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "be4ef7d7f1f4743a5048cab6a4dced5e5d1e5b14", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2021-11-20T05:52:51.224 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "be4ef7d7f1f4743a5048cab6a4dced5e5d1e5b14", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2021-11-20T05:52:51.392 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "5a66a2928a1d64d8d11230f172eb6b696c57c58c", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2021-11-20T05:52:51.664 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "5a66a2928a1d64d8d11230f172eb6b696c57c58c", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2021-11-20T05:52:51.821 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "12b6934a87ad915f812630b0a7e54663770cd059", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2021-11-20T05:52:52.101 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "12b6934a87ad915f812630b0a7e54663770cd059", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2021-11-20T05:52:52.310 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "c36ac65b18cdf8071cdb7bd4e46c5639be96cbac", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "md5sum": "76ff786610d5a5d6d02f2e8fd5311b89", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387571.8520246-28579-101143396920677/source", "state": "file", "uid": 0} 2021-11-20T05:52:52.332 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:52 +0000 (0:00:01.574) 0:03:52.108 ***** 2021-11-20T05:52:52.593 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "c36ac65b18cdf8071cdb7bd4e46c5639be96cbac", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "md5sum": "76ff786610d5a5d6d02f2e8fd5311b89", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387572.1298077-28608-83529829547525/source", "state": "file", "uid": 0} 2021-11-20T05:52:52.619 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:52 +0000 (0:00:00.286) 0:03:52.394 ***** 2021-11-20T05:52:52.814 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-11-20T05:52:52.816 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-11-20T05:52:53.108 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-11-20T05:52:53.309 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'apache-arrow', 'value': {'name': 'Apache Arrow', 'baseurl': 'https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch', 'enabled': 1, 'gpgcheck': 1}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "86b924db6e383ddad921a207c09a708e6cca6a67", "dest": "/etc/yum.repos.d/apache-arrow.repo", "gid": 0, "group": "root", "item": {"key": "apache-arrow", "value": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}}, "md5sum": "183d0889f7213a1e38c7f48965d7549e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 215, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387572.8460693-28720-37078196403280/source", "state": "file", "uid": 0} 2021-11-20T05:52:53.335 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:53 +0000 (0:00:00.716) 0:03:53.111 ***** 2021-11-20T05:52:53.604 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'apache-arrow', 'value': {'name': 'Apache Arrow', 'baseurl': 'https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch', 'enabled': 1, 'gpgcheck': 1}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "86b924db6e383ddad921a207c09a708e6cca6a67", "dest": "/etc/yum.repos.d/apache-arrow.repo", "gid": 0, "group": "root", "item": {"key": "apache-arrow", "value": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}}, "md5sum": "183d0889f7213a1e38c7f48965d7549e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 215, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387573.1440706-28743-243050422735877/source", "state": "file", "uid": 0} 2021-11-20T05:52:53.629 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:53 +0000 (0:00:00.293) 0:03:53.404 ***** 2021-11-20T05:52:53.819 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-11-20T05:52:53.821 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-11-20T05:52:54.102 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-11-20T05:52:54.270 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-11-20T05:52:54.561 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-11-20T05:52:54.699 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-11-20T05:52:55.014 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-11-20T05:52:55.194 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "md5sum": "d96965513d5dae48e1f179ee52900b67", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387574.7375903-28816-46509021554151/source", "state": "file", "uid": 0} 2021-11-20T05:52:55.233 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:55 +0000 (0:00:01.603) 0:03:55.008 ***** 2021-11-20T05:52:55.505 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "md5sum": "d96965513d5dae48e1f179ee52900b67", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387575.039601-28839-69196706211414/source", "state": "file", "uid": 0} 2021-11-20T05:52:55.532 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:55 +0000 (0:00:00.299) 0:03:55.307 ***** 2021-11-20T05:52:55.708 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-11-20T05:52:55.710 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.266753", "end": "2021-11-20 05:52:55.691383", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-11-20 05:52:55.424630", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'comma 2021-11-20T05:52:55.711 INFO:teuthology.task.ansible.out:nd_warnings=False' in ansible.cfg to get rid of this message. 2021-11-20T05:52:55.733 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:55 +0000 (0:00:00.201) 0:03:55.509 ***** 2021-11-20T05:52:56.007 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.277074", "end": "2021-11-20 05:52:55.999356", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-11-20 05:52:55.722282", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2021-11-20T05:52:56.042 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:52:56 +0000 (0:00:00.309) 0:03:55.818 ***** 2021-11-20T05:53:03.204 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-11-20T05:53:03.206 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:53:03.228 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:03 +0000 (0:00:07.185) 0:04:03.004 ***** 2021-11-20T05:53:03.236 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:53:03.263 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:03 +0000 (0:00:00.034) 0:04:03.038 ***** 2021-11-20T05:53:04.006 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-11-20T05:53:04.008 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/release.asc"} 2021-11-20T05:53:04.038 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/release.asc"} 2021-11-20T05:53:04.474 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/autobuild.asc"} 2021-11-20T05:53:04.486 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/autobuild.asc"} 2021-11-20T05:53:05.791 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=https://dist.apache.org/repos/dist/dev/arrow/KEYS) => {"ansible_loop_var": "item", "changed": true, "item": "https://dist.apache.org/repos/dist/dev/arrow/KEYS"} 2021-11-20T05:53:05.819 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:05 +0000 (0:00:02.555) 0:04:05.593 ***** 2021-11-20T05:53:05.930 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=https://dist.apache.org/repos/dist/dev/arrow/KEYS) => {"ansible_loop_var": "item", "changed": true, "item": "https://dist.apache.org/repos/dist/dev/arrow/KEYS"} 2021-11-20T05:53:05.955 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:05 +0000 (0:00:00.137) 0:04:05.730 ***** 2021-11-20T05:53:06.216 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** [WARNING]: Consider using the yum module rather than running 'yum'. If you need to use command because yum is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.191465", "end": "2021-11-20 05:53:06.207024", "rc": 0, "start": "2021-11-20 05:53:06.015559", "stderr": "", "stderr_lines": [], "stdout": "57 files removed", "stdout_lines": ["57 files removed"]} 2021-11-20T05:53:06.239 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:06 +0000 (0:00:00.283) 0:04:06.014 ***** 2021-11-20T05:53:06.277 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:53:06.279 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-11-20T05:53:06.300 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:06 +0000 (0:00:00.060) 0:04:06.075 ***** 2021-11-20T05:53:06.339 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:06.359 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:06 +0000 (0:00:00.059) 0:04:06.135 ***** 2021-11-20T05:53:06.366 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-11-20T05:53:06.367 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.182468", "end": "2021-11-20 05:53:06.335892", "rc": 0, "start": "2021-11-20 05:53:06.153424", "stderr": "", "stderr_lines": [], "stdout": "57 files removed", "stdout_lines": ["57 files removed"]} 2021-11-20T05:53:06.391 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:06 +0000 (0:00:00.031) 0:04:06.166 ***** 2021-11-20T05:53:06.398 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-11-20T05:53:06.399 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:06.423 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:06 +0000 (0:00:00.031) 0:04:06.198 ***** 2021-11-20T05:53:06.439 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:53:06.441 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-11-20T05:53:06.463 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:06 +0000 (0:00:00.040) 0:04:06.238 ***** 2021-11-20T05:53:06.500 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-11-20T05:53:06.502 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:06.522 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:06 +0000 (0:00:00.059) 0:04:06.298 ***** 2021-11-20T05:53:06.560 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-11-20T05:53:06.562 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:06.589 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:06 +0000 (0:00:00.066) 0:04:06.364 ***** 2021-11-20T05:53:06.640 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-11-20T05:53:06.643 INFO:teuthology.task.ansible.out:fatal: [smithi178.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007498", "end": "2021-11-20 05:53:06.625198", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 05:53:06.617700", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-11-20T05:53:06.669 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:06 +0000 (0:00:00.079) 0:04:06.444 ***** 2021-11-20T05:53:06.709 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:06.740 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:06 +0000 (0:00:00.071) 0:04:06.515 ***** 2021-11-20T05:53:06.789 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-11-20T05:53:06.791 INFO:teuthology.task.ansible.out:fatal: [smithi139.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007314", "end": "2021-11-20 05:53:06.780083", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 05:53:06.772769", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-11-20T05:53:06.818 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:06 +0000 (0:00:00.075) 0:04:06.591 ***** 2021-11-20T05:53:06.855 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:06.877 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:06 +0000 (0:00:00.061) 0:04:06.653 ***** 2021-11-20T05:53:25.015 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-11-20T05:53:25.017 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-11-20T05:53:25.038 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:25 +0000 (0:00:18.160) 0:04:24.813 ***** 2021-11-20T05:53:25.046 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-11-20T05:53:25.071 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:25 +0000 (0:00:00.033) 0:04:24.847 ***** 2021-11-20T05:53:28.092 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-11-20T05:53:28.094 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-11-20T05:53:28.123 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:28 +0000 (0:00:03.050) 0:04:27.897 ***** 2021-11-20T05:53:28.227 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-11-20T05:53:28.251 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:28 +0000 (0:00:00.129) 0:04:28.027 ***** 2021-11-20T05:53:37.084 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T05:53:37.086 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: qemu-kvm-common-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: python3-dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: qemu-kvm-core-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: yum-utils-4.0.21-3.el8.noarch", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: python3-dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: yum-utils-4.0.18-4.el8.noarch"]} 2021-11-20T05:53:37.113 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:37 +0000 (0:00:08.862) 0:04:36.889 ***** 2021-11-20T05:53:37.154 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-11-20T05:53:37.156 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:37.177 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:37 +0000 (0:00:00.063) 0:04:36.953 ***** 2021-11-20T05:53:37.491 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: qemu-kvm-common-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: python3-dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: qemu-kvm-core-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: yum-utils-4.0.21-3.el8.noarch", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: python3-dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: yum-utils-4.0.18-4.el8.noarch"]} 2021-11-20T05:53:37.514 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:37 +0000 (0:00:00.336) 0:04:37.290 ***** 2021-11-20T05:53:37.552 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-11-20T05:53:37.554 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:37.577 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:37 +0000 (0:00:00.063) 0:04:37.353 ***** 2021-11-20T05:53:39.324 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-11-20T05:53:39.325 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:53:39.348 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:39 +0000 (0:00:01.770) 0:04:39.123 ***** 2021-11-20T05:53:39.629 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:53:39.652 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:39 +0000 (0:00:00.304) 0:04:39.427 ***** 2021-11-20T05:53:47.063 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-20T05:53:47.065 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: systemd-pam-239-51.el8.x86_64", "Installed: systemd-239-51.el8.x86_64", "Installed: systemd-libs-239-51.el8.x86_64", "Installed: systemd-udev-239-51.el8.x86_64", "Removed: systemd-239-45.el8.x86_64", "Removed: systemd-libs-239-45.el8.x86_64", "Removed: systemd-pam-239-45.el8.x86_64", "Removed: systemd-udev-239-45.el8.x86_64"]} 2021-11-20T05:53:47.089 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:47 +0000 (0:00:07.436) 0:04:46.864 ***** 2021-11-20T05:53:47.413 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-11-20T05:53:47.414 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service dbus.service basic.target dbus.socket sysinit.target system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ipset.service shutdown.target iptables.service nftables.service ip6tables.service ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate" 2021-11-20T05:53:47.416 INFO:teuthology.task.ansible.out:: "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": 2021-11-20T05:53:47.418 INFO:teuthology.task.ansible.out:"0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket sysinit.target system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "nul 2021-11-20T05:53:47.419 INFO:teuthology.task.ansible.out:l", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-11-20T05:53:47.437 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:47 +0000 (0:00:00.348) 0:04:47.213 ***** 2021-11-20T05:53:47.726 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi139.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} 2021-11-20T05:53:47.727 INFO:teuthology.task.ansible.out:...ignoring 2021-11-20T05:53:47.748 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:47 +0000 (0:00:00.310) 0:04:47.524 ***** 2021-11-20T05:53:47.807 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-20T05:53:47.809 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: systemd-pam-239-51.el8.x86_64", "Installed: systemd-239-51.el8.x86_64", "Installed: systemd-libs-239-51.el8.x86_64", "Installed: systemd-udev-239-51.el8.x86_64", "Removed: systemd-239-45.el8.x86_64", "Removed: systemd-libs-239-45.el8.x86_64", "Removed: systemd-pam-239-45.el8.x86_64", "Removed: systemd-udev-239-45.el8.x86_64"]} 2021-11-20T05:53:47.832 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:47 +0000 (0:00:00.083) 0:04:47.607 ***** 2021-11-20T05:53:48.142 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-11-20T05:53:48.143 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "sysinit.target dbus.socket polkit.service system.slice dbus.service basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ip6tables.service shutdown.target ebtables.service nftables.service ipset.service iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate" 2021-11-20T05:53:48.145 INFO:teuthology.task.ansible.out:: "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": 2021-11-20T05:53:48.147 INFO:teuthology.task.ansible.out:"0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "nul 2021-11-20T05:53:48.149 INFO:teuthology.task.ansible.out:l", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203587", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-11-20T05:53:48.167 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.335) 0:04:47.943 ***** 2021-11-20T05:53:48.378 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-11-20T05:53:48.380 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-11-20T05:53:48.402 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.234) 0:04:48.177 ***** 2021-11-20T05:53:48.430 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-11-20T05:53:48.432 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:48.452 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.050) 0:04:48.228 ***** 2021-11-20T05:53:48.470 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-11-20T05:53:48.472 INFO:teuthology.task.ansible.out:fatal: [smithi178.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-11-20T05:53:48.494 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.041) 0:04:48.269 ***** 2021-11-20T05:53:48.501 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-11-20T05:53:48.503 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:48.526 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.032) 0:04:48.301 ***** 2021-11-20T05:53:48.553 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-11-20T05:53:48.556 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:48.577 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.050) 0:04:48.352 ***** 2021-11-20T05:53:48.604 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-11-20T05:53:48.606 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:48.629 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.051) 0:04:48.404 ***** 2021-11-20T05:53:48.656 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-11-20T05:53:48.658 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:48.679 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.049) 0:04:48.454 ***** 2021-11-20T05:53:48.706 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-11-20T05:53:48.707 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:48.728 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.049) 0:04:48.503 ***** 2021-11-20T05:53:48.755 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-11-20T05:53:48.777 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.049) 0:04:48.553 ***** 2021-11-20T05:53:48.784 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-11-20T05:53:48.786 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-11-20T05:53:48.808 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.031) 0:04:48.584 ***** 2021-11-20T05:53:48.816 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-11-20T05:53:48.819 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:48.841 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.032) 0:04:48.616 ***** 2021-11-20T05:53:48.848 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-11-20T05:53:48.849 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:48.871 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.030) 0:04:48.647 ***** 2021-11-20T05:53:48.877 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-11-20T05:53:48.881 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:48.903 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.031) 0:04:48.679 ***** 2021-11-20T05:53:48.912 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-11-20T05:53:48.913 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:48.936 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.032) 0:04:48.711 ***** 2021-11-20T05:53:48.944 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-11-20T05:53:48.946 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-11-20T05:53:48.973 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:48 +0000 (0:00:00.037) 0:04:48.748 ***** 2021-11-20T05:53:48.980 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-11-20T05:53:48.982 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.004 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.031) 0:04:48.780 ***** 2021-11-20T05:53:49.010 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-11-20T05:53:49.012 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.034 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.030) 0:04:48.810 ***** 2021-11-20T05:53:49.042 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-11-20T05:53:49.044 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.066 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.031) 0:04:48.841 ***** 2021-11-20T05:53:49.071 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-11-20T05:53:49.096 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.029) 0:04:48.871 ***** 2021-11-20T05:53:49.102 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.126 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.029) 0:04:48.901 ***** 2021-11-20T05:53:49.132 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-11-20T05:53:49.133 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.162 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.036) 0:04:48.938 ***** 2021-11-20T05:53:49.169 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-11-20T05:53:49.171 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.194 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.030) 0:04:48.969 ***** 2021-11-20T05:53:49.199 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-11-20T05:53:49.202 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.224 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.030) 0:04:48.999 ***** 2021-11-20T05:53:49.231 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-11-20T05:53:49.255 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.031) 0:04:49.031 ***** 2021-11-20T05:53:49.261 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-11-20T05:53:49.263 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.285 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.030) 0:04:49.061 ***** 2021-11-20T05:53:49.292 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-11-20T05:53:49.293 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.315 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.029) 0:04:49.090 ***** 2021-11-20T05:53:49.321 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-20T05:53:49.323 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.350 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.034) 0:04:49.125 ***** 2021-11-20T05:53:49.358 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-11-20T05:53:49.359 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.381 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.031) 0:04:49.157 ***** 2021-11-20T05:53:49.388 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-11-20T05:53:49.389 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.412 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.030) 0:04:49.188 ***** 2021-11-20T05:53:49.421 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-11-20T05:53:49.423 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.459 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.046) 0:04:49.235 ***** 2021-11-20T05:53:49.466 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T05:53:49.467 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.490 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.031) 0:04:49.266 ***** 2021-11-20T05:53:49.497 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-11-20T05:53:49.499 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.526 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.030) 0:04:49.296 ***** 2021-11-20T05:53:49.527 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-11-20T05:53:49.528 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.551 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.030) 0:04:49.327 ***** 2021-11-20T05:53:49.558 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-11-20T05:53:49.582 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.030) 0:04:49.357 ***** 2021-11-20T05:53:49.588 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-11-20T05:53:49.589 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.612 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.029) 0:04:49.387 ***** 2021-11-20T05:53:49.618 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-11-20T05:53:49.620 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.642 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.030) 0:04:49.418 ***** 2021-11-20T05:53:49.649 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-11-20T05:53:49.651 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.673 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.030) 0:04:49.448 ***** 2021-11-20T05:53:49.680 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-11-20T05:53:49.682 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.707 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.033) 0:04:49.482 ***** 2021-11-20T05:53:49.713 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-11-20T05:53:49.714 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.736 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.029) 0:04:49.512 ***** 2021-11-20T05:53:49.743 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-11-20T05:53:49.745 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.768 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.031) 0:04:49.543 ***** 2021-11-20T05:53:49.775 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-11-20T05:53:49.776 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.798 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.030) 0:04:49.573 ***** 2021-11-20T05:53:49.806 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-20T05:53:49.808 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.830 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.031) 0:04:49.605 ***** 2021-11-20T05:53:49.835 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-11-20T05:53:49.837 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.858 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.028) 0:04:49.634 ***** 2021-11-20T05:53:49.864 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-11-20T05:53:49.866 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.889 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.030) 0:04:49.664 ***** 2021-11-20T05:53:49.896 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-11-20T05:53:49.897 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.921 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.032) 0:04:49.696 ***** 2021-11-20T05:53:49.930 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T05:53:49.932 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.954 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.033) 0:04:49.730 ***** 2021-11-20T05:53:49.961 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-11-20T05:53:49.962 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:49.984 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:49 +0000 (0:00:00.030) 0:04:49.760 ***** 2021-11-20T05:53:49.990 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-11-20T05:53:49.992 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.014 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.029) 0:04:49.789 ***** 2021-11-20T05:53:50.020 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-11-20T05:53:50.022 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.044 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.029) 0:04:49.819 ***** 2021-11-20T05:53:50.050 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-11-20T05:53:50.052 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.075 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.030) 0:04:49.850 ***** 2021-11-20T05:53:50.081 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-20T05:53:50.082 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.105 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.030) 0:04:49.881 ***** 2021-11-20T05:53:50.112 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-11-20T05:53:50.114 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.138 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.032) 0:04:49.913 ***** 2021-11-20T05:53:50.145 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-11-20T05:53:50.147 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.169 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.031) 0:04:49.945 ***** 2021-11-20T05:53:50.176 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-11-20T05:53:50.178 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.201 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.031) 0:04:49.976 ***** 2021-11-20T05:53:50.207 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-11-20T05:53:50.209 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.231 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.030) 0:04:50.006 ***** 2021-11-20T05:53:50.238 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-11-20T05:53:50.239 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.262 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.030) 0:04:50.037 ***** 2021-11-20T05:53:50.267 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-11-20T05:53:50.269 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.294 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.032) 0:04:50.070 ***** 2021-11-20T05:53:50.302 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-11-20T05:53:50.303 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.328 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.033) 0:04:50.103 ***** 2021-11-20T05:53:50.335 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-11-20T05:53:50.337 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.359 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.030) 0:04:50.134 ***** 2021-11-20T05:53:50.366 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.390 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.030) 0:04:50.165 ***** 2021-11-20T05:53:50.396 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-20T05:53:50.398 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.421 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.031) 0:04:50.197 ***** 2021-11-20T05:53:50.428 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-11-20T05:53:50.430 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.455 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.034) 0:04:50.231 ***** 2021-11-20T05:53:50.462 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-11-20T05:53:50.463 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.485 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.029) 0:04:50.261 ***** 2021-11-20T05:53:50.492 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-11-20T05:53:50.493 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.677 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.192) 0:04:50.453 ***** 2021-11-20T05:53:50.684 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-11-20T05:53:50.686 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.708 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.030) 0:04:50.484 ***** 2021-11-20T05:53:50.715 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-20T05:53:50.717 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.739 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.030) 0:04:50.514 ***** 2021-11-20T05:53:50.746 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:53:50.747 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.770 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.031) 0:04:50.546 ***** 2021-11-20T05:53:50.777 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-11-20T05:53:50.779 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.802 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.031) 0:04:50.577 ***** 2021-11-20T05:53:50.809 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-11-20T05:53:50.811 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.833 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.031) 0:04:50.609 ***** 2021-11-20T05:53:50.840 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-11-20T05:53:50.842 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.864 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.030) 0:04:50.639 ***** 2021-11-20T05:53:50.870 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-20T05:53:50.872 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.894 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.030) 0:04:50.670 ***** 2021-11-20T05:53:50.901 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-11-20T05:53:50.903 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.925 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.030) 0:04:50.700 ***** 2021-11-20T05:53:50.932 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.956 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.031) 0:04:50.732 ***** 2021-11-20T05:53:50.964 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-11-20T05:53:50.966 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:50.989 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:50 +0000 (0:00:00.032) 0:04:50.764 ***** 2021-11-20T05:53:50.996 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-11-20T05:53:50.998 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:51.020 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.030) 0:04:50.795 ***** 2021-11-20T05:53:51.027 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-20T05:53:51.028 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:51.051 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.031) 0:04:50.827 ***** 2021-11-20T05:53:51.079 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-11-20T05:53:51.081 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:51.102 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.050) 0:04:50.878 ***** 2021-11-20T05:53:51.131 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-11-20T05:53:51.133 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:51.153 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.051) 0:04:50.929 ***** 2021-11-20T05:53:51.181 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:53:51.183 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:51.203 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.049) 0:04:50.979 ***** 2021-11-20T05:53:51.231 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T05:53:51.233 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} TASK [testnode : Remove packages] ********************************************** skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:51.255 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.051) 0:04:51.030 ***** 2021-11-20T05:53:51.276 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.021) 0:04:51.052 ***** 2021-11-20T05:53:51.305 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:51.329 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.052) 0:04:51.104 ***** 2021-11-20T05:53:51.356 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-20T05:53:51.358 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:51.379 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.050) 0:04:51.155 ***** 2021-11-20T05:53:51.407 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-11-20T05:53:51.408 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:51.435 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.056) 0:04:51.211 ***** 2021-11-20T05:53:51.464 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T05:53:51.466 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:53:51.490 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.054) 0:04:51.265 ***** 2021-11-20T05:53:51.653 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T05:53:51.655 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-11-20T05:53:51.678 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.188) 0:04:51.453 ***** 2021-11-20T05:53:51.695 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T05:53:51.697 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-11-20T05:53:51.719 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.041) 0:04:51.495 ***** 2021-11-20T05:53:51.747 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-20T05:53:51.749 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:51.770 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.050) 0:04:51.545 ***** 2021-11-20T05:53:51.800 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-20T05:53:51.801 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:51.822 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.052) 0:04:51.598 ***** 2021-11-20T05:53:51.861 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-11-20T05:53:51.863 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:51.885 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.062) 0:04:51.660 ***** 2021-11-20T05:53:51.892 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T05:53:51.893 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:53:51.927 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.041) 0:04:51.702 ***** 2021-11-20T05:53:51.936 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T05:53:51.938 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:51.962 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:51 +0000 (0:00:00.035) 0:04:51.738 ***** 2021-11-20T05:53:51.990 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T05:53:51.992 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.017 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.050) 0:04:51.788 ***** 2021-11-20T05:53:52.040 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.064 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.051) 0:04:51.840 ***** 2021-11-20T05:53:52.092 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T05:53:52.093 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.114 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.050) 0:04:51.890 ***** 2021-11-20T05:53:52.133 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T05:53:52.135 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-11-20T05:53:52.156 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.041) 0:04:51.932 ***** 2021-11-20T05:53:52.163 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T05:53:52.165 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.187 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.030) 0:04:51.962 ***** 2021-11-20T05:53:52.195 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-20T05:53:52.196 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.220 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.032) 0:04:51.995 ***** 2021-11-20T05:53:52.227 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T05:53:52.228 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.251 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.030) 0:04:52.026 ***** 2021-11-20T05:53:52.258 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-20T05:53:52.260 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.282 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.030) 0:04:52.057 ***** 2021-11-20T05:53:52.289 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-11-20T05:53:52.290 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.319 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.037) 0:04:52.095 ***** 2021-11-20T05:53:52.326 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-11-20T05:53:52.328 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.353 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.033) 0:04:52.128 ***** 2021-11-20T05:53:52.359 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-11-20T05:53:52.360 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.382 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.029) 0:04:52.158 ***** 2021-11-20T05:53:52.389 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T05:53:52.391 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.414 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.031) 0:04:52.189 ***** 2021-11-20T05:53:52.421 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-20T05:53:52.422 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.444 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.030) 0:04:52.220 ***** 2021-11-20T05:53:52.450 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T05:53:52.452 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.474 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.029) 0:04:52.250 ***** 2021-11-20T05:53:52.481 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-20T05:53:52.483 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.505 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.030) 0:04:52.280 ***** 2021-11-20T05:53:52.511 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T05:53:52.513 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.536 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.031) 0:04:52.311 ***** 2021-11-20T05:53:52.542 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-11-20T05:53:52.544 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.566 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.030) 0:04:52.342 ***** 2021-11-20T05:53:52.573 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-20T05:53:52.574 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.596 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.030) 0:04:52.372 ***** 2021-11-20T05:53:52.603 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-11-20T05:53:52.605 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.626 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.029) 0:04:52.402 ***** 2021-11-20T05:53:52.633 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-20T05:53:52.635 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.656 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.030) 0:04:52.432 ***** 2021-11-20T05:53:52.663 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-11-20T05:53:52.666 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.687 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.030) 0:04:52.462 ***** 2021-11-20T05:53:52.693 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-20T05:53:52.695 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.717 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.030) 0:04:52.492 ***** 2021-11-20T05:53:52.724 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.747 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.030) 0:04:52.523 ***** 2021-11-20T05:53:52.754 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-11-20T05:53:52.756 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.778 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.030) 0:04:52.553 ***** 2021-11-20T05:53:52.784 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-11-20T05:53:52.785 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.807 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.029) 0:04:52.582 ***** 2021-11-20T05:53:52.813 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-11-20T05:53:52.815 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.838 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.031) 0:04:52.614 ***** 2021-11-20T05:53:52.845 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-11-20T05:53:52.847 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.870 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.030) 0:04:52.645 ***** 2021-11-20T05:53:52.876 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-20T05:53:52.878 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.901 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.031) 0:04:52.676 ***** 2021-11-20T05:53:52.907 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-11-20T05:53:52.909 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.931 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.030) 0:04:52.706 ***** 2021-11-20T05:53:52.938 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.961 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.030) 0:04:52.737 ***** 2021-11-20T05:53:52.968 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-11-20T05:53:52.970 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:52.993 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:52 +0000 (0:00:00.031) 0:04:52.768 ***** 2021-11-20T05:53:53.000 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.024 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.030) 0:04:52.799 ***** 2021-11-20T05:53:53.030 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-11-20T05:53:53.032 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.054 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.030) 0:04:52.829 ***** 2021-11-20T05:53:53.060 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-11-20T05:53:53.062 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.088 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.034) 0:04:52.863 ***** 2021-11-20T05:53:53.104 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:53:53.106 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-11-20T05:53:53.128 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.039) 0:04:52.903 ***** 2021-11-20T05:53:53.136 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-11-20T05:53:53.138 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.161 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.032) 0:04:52.936 ***** 2021-11-20T05:53:53.179 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-11-20T05:53:53.180 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} ok: [smithi139.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-11-20T05:53:53.191 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.193 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi139.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.216 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.055) 0:04:52.992 ***** 2021-11-20T05:53:53.224 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.225 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.251 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.035) 0:04:53.027 ***** 2021-11-20T05:53:53.258 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-11-20T05:53:53.260 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.283 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.031) 0:04:53.059 ***** 2021-11-20T05:53:53.312 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.334 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.050) 0:04:53.109 ***** 2021-11-20T05:53:53.362 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.386 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.052) 0:04:53.161 ***** 2021-11-20T05:53:53.414 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-11-20T05:53:53.416 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.450 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.062) 0:04:53.223 ***** 2021-11-20T05:53:53.470 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-11-20T05:53:53.471 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003467", "end": "2021-11-20 05:53:53.425772", "rc": 0, "start": "2021-11-20 05:53:53.422305", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock"]} 2021-11-20T05:53:53.495 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.046) 0:04:53.270 ***** 2021-11-20T05:53:53.502 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-11-20T05:53:53.503 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.525 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.030) 0:04:53.301 ***** 2021-11-20T05:53:53.530 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-11-20T05:53:53.532 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.557 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.031) 0:04:53.332 ***** 2021-11-20T05:53:53.563 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:53:53.565 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-11-20T05:53:53.588 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.030) 0:04:53.362 ***** 2021-11-20T05:53:53.593 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-11-20T05:53:53.595 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.618 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.030) 0:04:53.393 ***** 2021-11-20T05:53:53.636 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-11-20T05:53:53.638 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} ok: [smithi178.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-11-20T05:53:53.648 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.649 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi178.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.671 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.053) 0:04:53.446 ***** 2021-11-20T05:53:53.678 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.679 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.704 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.033) 0:04:53.480 ***** 2021-11-20T05:53:53.711 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-11-20T05:53:53.713 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.736 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.031) 0:04:53.511 ***** 2021-11-20T05:53:53.909 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-11-20T05:53:53.911 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003369", "end": "2021-11-20 05:53:53.879772", "rc": 0, "start": "2021-11-20 05:53:53.876403", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock"]} 2021-11-20T05:53:53.935 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.199) 0:04:53.711 ***** 2021-11-20T05:53:53.963 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-11-20T05:53:53.965 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:53.996 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:53 +0000 (0:00:00.061) 0:04:53.772 ***** 2021-11-20T05:53:54.023 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-11-20T05:53:54.026 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:54.046 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:54 +0000 (0:00:00.049) 0:04:53.821 ***** 2021-11-20T05:53:54.072 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-11-20T05:53:54.074 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:54.094 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:54 +0000 (0:00:00.048) 0:04:53.870 ***** 2021-11-20T05:53:54.121 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-11-20T05:53:54.122 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:54.143 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:54 +0000 (0:00:00.048) 0:04:53.918 ***** 2021-11-20T05:53:55.725 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:53:55.750 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:55 +0000 (0:00:01.606) 0:04:55.525 ***** 2021-11-20T05:53:55.799 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-20T05:53:55.801 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140164096, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085001, 'block_used': 13010107, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085001, "block_size": 4096, "block_total": 240095108, "block_used": 13010107, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140164096, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}} 2021-11-20T05:53:55.823 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:55 +0000 (0:00:00.074) 0:04:55.599 ***** 2021-11-20T05:53:56.018 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-11-20T05:53:56.020 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005229", "end": "2021-11-20 05:53:56.012166", "rc": 0, "start": "2021-11-20 05:53:56.006937", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:53:56.043 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:56 +0000 (0:00:00.219) 0:04:55.818 ***** 2021-11-20T05:53:56.092 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-11-20T05:53:56.093 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140164096, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085001, 'block_used': 13010107, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085001, "block_size": 4096, "block_total": 240095108, "block_used": 13010107, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140164096, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:56.115 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:56 +0000 (0:00:00.072) 0:04:55.891 ***** 2021-11-20T05:53:56.133 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:53:56.158 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:56 +0000 (0:00:00.042) 0:04:55.934 ***** 2021-11-20T05:53:56.165 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-11-20T05:53:56.190 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:56 +0000 (0:00:00.031) 0:04:55.965 ***** 2021-11-20T05:53:56.207 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-20T05:53:56.209 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140180480, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085005, 'block_used': 13010103, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085005, "block_size": 4096, "block_total": 240095108, "block_used": 13010103, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140180480, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}} 2021-11-20T05:53:56.233 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:56 +0000 (0:00:00.043) 0:04:56.008 ***** 2021-11-20T05:53:56.431 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-11-20T05:53:56.432 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005118", "end": "2021-11-20 05:53:56.422354", "rc": 0, "start": "2021-11-20 05:53:56.417236", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:53:56.457 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:56 +0000 (0:00:00.222) 0:04:56.230 ***** 2021-11-20T05:53:56.494 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-11-20T05:53:56.496 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140180480, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085005, 'block_used': 13010103, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085005, "block_size": 4096, "block_total": 240095108, "block_used": 13010103, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140180480, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:53:56.517 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:56 +0000 (0:00:00.062) 0:04:56.293 ***** 2021-11-20T05:53:56.546 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-11-20T05:53:56.571 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:53:56 +0000 (0:00:00.053) 0:04:56.346 ***** 2021-11-20T05:53:57.742 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-20T05:53:57.743 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5lIg590hoG09Ew4rDlFGbST5BCQskQTaH'], 'uuids': ['1dd2667b-c647-4c00-b0c9-6d65a0b68cff'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.334011", "end": "2021-11-20 05:53:57.725016", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5lIg590hoG09Ew4rDlFGbST5BCQskQTaH"], "labels": [], "masters": [], "uuids": ["1dd2667b-c647-4c00-b0c9-6d65a0b68cff"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:53:56.391005", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:53:58.366 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBdF5Bbs5E2qcOiZFR0UrtCcaiDHeMLkiBp'], 'uuids': ['e0efb917-b404-4818-ae2d-351782cf5117'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.578612", "end": "2021-11-20 05:53:58.349938", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBdF5Bbs5E2qcOiZFR0UrtCcaiDHeMLkiBp"], "labels": [], "masters": [], "uuids": ["e0efb917-b404-4818-ae2d-351782cf5117"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:53:56.771326", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:53:58.970 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-hW8iL9-Fsjt-NdUw-jlHY-7Fqu-WjR8-AdCJKL', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500N6400BGN', 'nvme-nvme.8086-504846543632303530304e3634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.038912", "end": "2021-11-20 05:53:58.954030", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-hW8iL9-Fsjt-NdUw-jlHY-7Fqu-WjR8-AdCJKL", "nvme-INTEL_SSDPEDMD400G4_PHFT620500N6400BGN", "nvme-nvme.8086-504846543632303530304e3634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:53:57.915118", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-11-20T05:53:58.972 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:53:59.619 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-5zYAGp-9Hr4-kkr8-CDDa-4zqB-9sKI-y0Hblv', 'nvme-INTEL_SSDPED1D480GA_PHMB75130029480DGN', 'nvme-nvme.8086-50484d42373531333030323934383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.063922", "end": "2021-11-20 05:53:59.604147", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-5zYAGp-9Hr4-kkr8-CDDa-4zqB-9sKI-y0Hblv", "nvme-INTEL_SSDPED1D480GA_PHMB75130029480DGN", "nvme-nvme.8086-50484d42373531333030323934383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:53:58.540225", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2021-11-20T05:53:59.621 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:54:00.231 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5e9ASkSsKnDPwfs1ebG5UPBXAwwmxZ0hU'], 'uuids': ['04851c9c-c700-4a11-b8b5-a625ada469be'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.065559", "end": "2021-11-20 05:54:00.212169", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5e9ASkSsKnDPwfs1ebG5UPBXAwwmxZ0hU"], "labels": [], "masters": [], "uuids": ["04851c9c-c700-4a11-b8b5-a625ada469be"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:53:59.146610", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:54:00.890 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBdDyfRg94VJ1oIcXL5YjRdG5vM6HKyxzJb'], 'uuids': ['6957f86f-c69f-4195-b737-005661be9bd5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.073304", "end": "2021-11-20 05:54:00.870365", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBdDyfRg94VJ1oIcXL5YjRdG5vM6HKyxzJb"], "labels": [], "masters": [], "uuids": ["6957f86f-c69f-4195-b737-005661be9bd5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:53:59.797061", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:54:01.472 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5P6PaTzqtMJCPOl331TcNoHg0BubjRlBs'], 'uuids': ['0e0fd0c5-0c2e-46f6-bee7-4d7ff449ac8c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.054140", "end": "2021-11-20 05:54:01.453063", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5P6PaTzqtMJCPOl331TcNoHg0BubjRlBs"], "labels": [], "masters": [], "uuids": ["0e0fd0c5-0c2e-46f6-bee7-4d7ff449ac8c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:00.398923", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:54:02.116 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBdPsMSmIhJVISPTg1TrVsLBOZXQfPgPyRR'], 'uuids': ['dbd5dbb4-ad47-46d6-8737-dc28b5200c36'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.038897", "end": "2021-11-20 05:54:02.095457", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBdPsMSmIhJVISPTg1TrVsLBOZXQfPgPyRR"], "labels": [], "masters": [], "uuids": ["dbd5dbb4-ad47-46d6-8737-dc28b5200c36"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:01.056560", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:54:02.721 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5wgx1FTYcljebGbQXXE8UftHVHeqPIofs'], 'uuids': ['85c86a21-108d-4965-b85c-664052699123'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.063183", "end": "2021-11-20 05:54:02.703901", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5wgx1FTYcljebGbQXXE8UftHVHeqPIofs"], "labels": [], "masters": [], "uuids": ["85c86a21-108d-4965-b85c-664052699123"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:01.640718", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:54:02.733 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFR4', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFR4', 'scsi-35000c500929226ed', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4', 'wwn-0x5000c500929226ed'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XFR4', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500929226ed', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFR4-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFR4-part1', 'scsi-35000c500929226ed-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4-part1', 'wwn-0x5000c500929226ed-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFR4", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFR4", "scsi-35000c500929226ed", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4", "wwn-0x5000c500929226ed"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFR4-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-20T05:54:02.735 INFO:teuthology.task.ansible.out:Z1W5XFR4-part1", "scsi-35000c500929226ed-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4-part1", "wwn-0x5000c500929226ed-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFR4", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500929226ed"}}, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:03.355 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBddvAVN2Mmjxe9aaNO4B0Ayh5IeL3aoSMK'], 'uuids': ['56f67c96-6ca6-4104-a3ac-f79a15e15829'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.053710", "end": "2021-11-20 05:54:03.337165", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBddvAVN2Mmjxe9aaNO4B0Ayh5IeL3aoSMK"], "labels": [], "masters": [], "uuids": ["56f67c96-6ca6-4104-a3ac-f79a15e15829"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:02.283455", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:54:03.367 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFJ4', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFJ4', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFJ4', 'scsi-35000c50092924753', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFJ4', 'wwn-0x5000c50092924753'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XFJ4', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50092924753', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFJ4-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFJ4-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFJ4-part1', 'scsi-35000c50092924753-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFJ4-part1', 'wwn-0x5000c50092924753-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFJ4", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFJ4", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFJ4", "scsi-35000c50092924753", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFJ4", "wwn-0x5000c50092924753"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFJ4-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFJ4-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-20T05:54:03.369 INFO:teuthology.task.ansible.out:Z1W5XFJ4-part1", "scsi-35000c50092924753-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFJ4-part1", "wwn-0x5000c50092924753-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFJ4", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092924753"}}, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:03.968 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5B3RJeOOgxx1ibVIcOw6rjowtQZsyOoMx'], 'uuids': ['b00ef35f-cf2c-4ddf-b2fa-ef6851b419ce'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.057216", "end": "2021-11-20 05:54:03.954054", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5B3RJeOOgxx1ibVIcOw6rjowtQZsyOoMx"], "labels": [], "masters": [], "uuids": ["b00ef35f-cf2c-4ddf-b2fa-ef6851b419ce"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:02.896838", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:54:04.009 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:04 +0000 (0:00:07.438) 0:05:03.784 ***** 2021-11-20T05:54:04.372 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-20T05:54:04.374 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5lIg590hoG09Ew4rDlFGbST5BCQskQTaH'], 'uuids': ['1dd2667b-c647-4c00-b0c9-6d65a0b68cff'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.139768", "end": "2021-11-20 05:54:04.357578", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5lIg590hoG09Ew4rDlFGbST5BCQskQTaH"], "labels": [], "masters": [], "uuids": ["1dd2667b-c647-4c00-b0c9-6d65a0b68cff"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:04.217810", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137125 s, 841 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137125 s, 841 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:04.581 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-hW8iL9-Fsjt-NdUw-jlHY-7Fqu-WjR8-AdCJKL', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500N6400BGN', 'nvme-nvme.8086-504846543632303530304e3634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.038538", "end": "2021-11-20 05:54:04.570394", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-hW8iL9-Fsjt-NdUw-jlHY-7Fqu-WjR8-AdCJKL", "nvme-INTEL_SSDPEDMD400G4_PHFT620500N6400BGN", "nvme-nvme.8086-504846543632303530304e3634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:04.531856", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0359135 s, 3.2 GB/s", "std 2021-11-20T05:54:04.583 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0359135 s, 3.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:04.645 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-20T05:54:04.647 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBd1aKS0340PJ372qrBqmTgMTkpyQLQPafC'], 'uuids': ['2ca7a29e-b662-4113-9681-92303bbb224b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.088411", "end": "2021-11-20 05:54:04.620126", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBd1aKS0340PJ372qrBqmTgMTkpyQLQPafC"], "labels": [], "masters": [], "uuids": ["2ca7a29e-b662-4113-9681-92303bbb224b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:03.531715", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-20T05:54:04.677 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:04 +0000 (0:00:00.667) 0:05:04.452 ***** 2021-11-20T05:54:04.912 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5e9ASkSsKnDPwfs1ebG5UPBXAwwmxZ0hU'], 'uuids': ['04851c9c-c700-4a11-b8b5-a625ada469be'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.152710", "end": "2021-11-20 05:54:04.902389", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5e9ASkSsKnDPwfs1ebG5UPBXAwwmxZ0hU"], "labels": [], "masters": [], "uuids": ["04851c9c-c700-4a11-b8b5-a625ada469be"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:04.749679", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150005 s, 769 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150005 s, 769 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:04.997 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBdF5Bbs5E2qcOiZFR0UrtCcaiDHeMLkiBp'], 'uuids': ['e0efb917-b404-4818-ae2d-351782cf5117'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.097676", "end": "2021-11-20 05:54:04.982899", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBdF5Bbs5E2qcOiZFR0UrtCcaiDHeMLkiBp"], "labels": [], "masters": [], "uuids": ["e0efb917-b404-4818-ae2d-351782cf5117"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:04.885223", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.095074 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.095074 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:05.207 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-5zYAGp-9Hr4-kkr8-CDDa-4zqB-9sKI-y0Hblv', 'nvme-INTEL_SSDPED1D480GA_PHMB75130029480DGN', 'nvme-nvme.8086-50484d42373531333030323934383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039000", "end": "2021-11-20 05:54:05.195171", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-5zYAGp-9Hr4-kkr8-CDDa-4zqB-9sKI-y0Hblv", "nvme-INTEL_SSDPED1D480GA_PHMB75130029480DGN", "nvme-nvme.8086-50484d42373531333030323934383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:05.156171", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0363082 s, 3.2 GB/s", "stderr_lines": [" 2021-11-20T05:54:05.209 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0363082 s, 3.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:05.229 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5P6PaTzqtMJCPOl331TcNoHg0BubjRlBs'], 'uuids': ['0e0fd0c5-0c2e-46f6-bee7-4d7ff449ac8c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.144513", "end": "2021-11-20 05:54:05.220809", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5P6PaTzqtMJCPOl331TcNoHg0BubjRlBs"], "labels": [], "masters": [], "uuids": ["0e0fd0c5-0c2e-46f6-bee7-4d7ff449ac8c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:05.076296", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.141798 s, 813 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.141798 s, 813 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:05.490 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBdDyfRg94VJ1oIcXL5YjRdG5vM6HKyxzJb'], 'uuids': ['6957f86f-c69f-4195-b737-005661be9bd5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.099525", "end": "2021-11-20 05:54:05.472985", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBdDyfRg94VJ1oIcXL5YjRdG5vM6HKyxzJb"], "labels": [], "masters": [], "uuids": ["6957f86f-c69f-4195-b737-005661be9bd5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:05.373460", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0967672 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0967672 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:05.554 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5wgx1FTYcljebGbQXXE8UftHVHeqPIofs'], 'uuids': ['85c86a21-108d-4965-b85c-664052699123'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.144106", "end": "2021-11-20 05:54:05.538000", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5wgx1FTYcljebGbQXXE8UftHVHeqPIofs"], "labels": [], "masters": [], "uuids": ["85c86a21-108d-4965-b85c-664052699123"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:05.393894", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.141493 s, 815 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.141493 s, 815 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:05.565 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFR4', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFR4', 'scsi-35000c500929226ed', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4', 'wwn-0x5000c500929226ed'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XFR4', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500929226ed', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFR4-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFR4-part1', 'scsi-35000c500929226ed-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4-part1', 'wwn-0x5000c500929226ed-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFR4", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFR4", "scsi-35000c500929226ed", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4", "wwn-0x5000c500929226ed"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFR4-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFR4-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-20T05:54:05.567 INFO:teuthology.task.ansible.out:Z1W5XFR4-part1", "scsi-35000c500929226ed-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFR4-part1", "wwn-0x5000c500929226ed-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFR4", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500929226ed"}}, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:05.765 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBdPsMSmIhJVISPTg1TrVsLBOZXQfPgPyRR'], 'uuids': ['dbd5dbb4-ad47-46d6-8737-dc28b5200c36'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.098055", "end": "2021-11-20 05:54:05.748117", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBdPsMSmIhJVISPTg1TrVsLBOZXQfPgPyRR"], "labels": [], "masters": [], "uuids": ["dbd5dbb4-ad47-46d6-8737-dc28b5200c36"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:05.650062", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0955137 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0955137 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:05.870 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5B3RJeOOgxx1ibVIcOw6rjowtQZsyOoMx'], 'uuids': ['b00ef35f-cf2c-4ddf-b2fa-ef6851b419ce'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.141206", "end": "2021-11-20 05:54:05.857595", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-l7keC1EtLBF8KrpRqf115fSChPu3fvx5B3RJeOOgxx1ibVIcOw6rjowtQZsyOoMx"], "labels": [], "masters": [], "uuids": ["b00ef35f-cf2c-4ddf-b2fa-ef6851b419ce"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:05.716389", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138637 s, 832 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138637 s, 832 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:05.899 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:05 +0000 (0:00:01.223) 0:05:05.675 ***** 2021-11-20T05:54:06.033 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBddvAVN2Mmjxe9aaNO4B0Ayh5IeL3aoSMK'], 'uuids': ['56f67c96-6ca6-4104-a3ac-f79a15e15829'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.098981", "end": "2021-11-20 05:54:06.023373", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBddvAVN2Mmjxe9aaNO4B0Ayh5IeL3aoSMK"], "labels": [], "masters": [], "uuids": ["56f67c96-6ca6-4104-a3ac-f79a15e15829"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:05.924392", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0963138 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0963138 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:06.045 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFJ4', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFJ4', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFJ4', 'scsi-35000c50092924753', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFJ4', 'wwn-0x5000c50092924753'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XFJ4', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50092924753', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFJ4-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFJ4-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFJ4-part1', 'scsi-35000c50092924753-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XFJ4-part1', 'wwn-0x5000c50092924753-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFJ4", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFJ4", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XFJ4", "scsi-35000c50092924753", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFJ4", "wwn-0x5000c50092924753"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFJ4-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XFJ4-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-20T05:54:06.047 INFO:teuthology.task.ansible.out:Z1W5XFJ4-part1", "scsi-35000c50092924753-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XFJ4-part1", "wwn-0x5000c50092924753-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFJ4", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092924753"}}, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:06.150 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-11-20T05:54:06.152 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.059317", "end": "2021-11-20 05:54:06.134104", "rc": 0, "start": "2021-11-20 05:54:06.074787", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:06.173 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:06 +0000 (0:00:00.273) 0:05:05.949 ***** 2021-11-20T05:54:06.316 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-20T05:54:06.317 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBd1aKS0340PJ372qrBqmTgMTkpyQLQPafC'], 'uuids': ['2ca7a29e-b662-4113-9681-92303bbb224b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.099666", "end": "2021-11-20 05:54:06.303626", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-QlpxB18ml4pdxWm8QAMu0K0GzKXE7cBd1aKS0340PJ372qrBqmTgMTkpyQLQPafC"], "labels": [], "masters": [], "uuids": ["2ca7a29e-b662-4113-9681-92303bbb224b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-20 05:54:06.203960", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0969566 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0969566 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:06.348 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:06 +0000 (0:00:00.174) 0:05:06.123 ***** 2021-11-20T05:54:06.397 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-11-20T05:54:06.398 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.044414", "end": "2021-11-20 05:54:06.390509", "rc": 0, "start": "2021-11-20 05:54:06.346095", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:06.429 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:06 +0000 (0:00:00.080) 0:05:06.204 ***** 2021-11-20T05:54:06.456 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-11-20T05:54:06.482 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:06 +0000 (0:00:00.053) 0:05:06.258 ***** 2021-11-20T05:54:06.510 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-11-20T05:54:06.511 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:06.535 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:06 +0000 (0:00:00.052) 0:05:06.311 ***** 2021-11-20T05:54:06.562 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:06.585 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:06 +0000 (0:00:00.049) 0:05:06.361 ***** 2021-11-20T05:54:06.613 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-11-20T05:54:06.614 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.069242", "end": "2021-11-20 05:54:06.592260", "rc": 0, "start": "2021-11-20 05:54:06.523018", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:06.638 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:06 +0000 (0:00:00.053) 0:05:06.414 ***** 2021-11-20T05:54:06.659 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:06 +0000 (0:00:00.020) 0:05:06.434 ***** 2021-11-20T05:54:06.676 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-11-20T05:54:06.678 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:06.701 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:06 +0000 (0:00:00.042) 0:05:06.476 ***** 2021-11-20T05:54:06.885 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-11-20T05:54:06.887 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.043225", "end": "2021-11-20 05:54:06.878444", "rc": 0, "start": "2021-11-20 05:54:06.835219", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:06.909 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:06 +0000 (0:00:00.207) 0:05:06.684 ***** 2021-11-20T05:54:06.915 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-11-20T05:54:06.917 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:54:06.941 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:06 +0000 (0:00:00.032) 0:05:06.717 ***** 2021-11-20T05:54:06.948 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-11-20T05:54:06.976 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:06 +0000 (0:00:00.034) 0:05:06.751 ***** 2021-11-20T05:54:06.993 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-20T05:54:06.995 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140164096, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085001, 'block_used': 13010107, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085001, "block_size": 4096, "block_total": 240095108, "block_used": 13010107, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140164096, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.018 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.041) 0:05:06.793 ***** 2021-11-20T05:54:07.026 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-11-20T05:54:07.027 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.052 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.034) 0:05:06.828 ***** 2021-11-20T05:54:07.059 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-11-20T05:54:07.060 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.086 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.033) 0:05:06.861 ***** 2021-11-20T05:54:07.094 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-11-20T05:54:07.096 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.118 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.031) 0:05:06.893 ***** 2021-11-20T05:54:07.124 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:54:07.126 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.148 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.030) 0:05:06.924 ***** 2021-11-20T05:54:07.155 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-11-20T05:54:07.157 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.182 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.033) 0:05:06.957 ***** 2021-11-20T05:54:07.188 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:54:07.190 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.213 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.031) 0:05:06.989 ***** 2021-11-20T05:54:07.220 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-11-20T05:54:07.222 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.244 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.031) 0:05:07.020 ***** 2021-11-20T05:54:07.251 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-11-20T05:54:07.252 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.276 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.032) 0:05:07.052 ***** 2021-11-20T05:54:07.315 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-11-20T05:54:07.316 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.337 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.060) 0:05:07.113 ***** 2021-11-20T05:54:07.365 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-11-20T05:54:07.366 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.387 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.049) 0:05:07.162 ***** 2021-11-20T05:54:07.435 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-11-20T05:54:07.437 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-20T05:54:07.459 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.071) 0:05:07.234 ***** 2021-11-20T05:54:07.507 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-20T05:54:07.509 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140180480, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227085005, 'block_used': 13010103, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227085005, "block_size": 4096, "block_total": 240095108, "block_used": 13010103, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140180480, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.530 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.071) 0:05:07.306 ***** 2021-11-20T05:54:07.558 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-11-20T05:54:07.560 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.580 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.049) 0:05:07.355 ***** 2021-11-20T05:54:07.607 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:54:07.609 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.629 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.049) 0:05:07.404 ***** 2021-11-20T05:54:07.655 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:54:07.657 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.678 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.048) 0:05:07.453 ***** 2021-11-20T05:54:07.705 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-11-20T05:54:07.706 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.727 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.049) 0:05:07.502 ***** 2021-11-20T05:54:07.754 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-11-20T05:54:07.755 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.776 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.048) 0:05:07.551 ***** 2021-11-20T05:54:07.802 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-11-20T05:54:07.804 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:07.828 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.052) 0:05:07.604 ***** 2021-11-20T05:54:07.908 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-11-20T05:54:07.910 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-11-20T05:54:07.932 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:07 +0000 (0:00:00.104) 0:05:07.708 ***** 2021-11-20T05:54:08.284 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-11-20T05:54:08.309 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:08 +0000 (0:00:00.376) 0:05:08.084 ***** 2021-11-20T05:54:08.401 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:54:08.733 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:54:08.744 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:54:09.078 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:54:09.412 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:54:09.423 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:54:09.746 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:54:09.771 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:09 +0000 (0:00:01.462) 0:05:09.546 ***** 2021-11-20T05:54:09.780 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:54:09.968 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-11-20T05:54:09.970 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-11-20T05:54:09.992 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:09 +0000 (0:00:00.221) 0:05:09.768 ***** 2021-11-20T05:54:10.104 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-11-20T05:54:10.106 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-20T05:54:10.129 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:10 +0000 (0:00:00.136) 0:05:09.904 ***** 2021-11-20T05:54:10.198 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-11-20T05:54:10.200 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:54:10.324 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-11-20T05:54:10.326 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-11-20T05:54:10.347 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:10 +0000 (0:00:00.218) 0:05:10.123 ***** 2021-11-20T05:54:10.374 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-11-20T05:54:10.376 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:54:10.556 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:54:10.567 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:54:10.724 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:54:10.738 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:10.762 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:10 +0000 (0:00:00.415) 0:05:10.538 ***** 2021-11-20T05:54:10.771 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:54:10.793 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:10.817 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:10 +0000 (0:00:00.054) 0:05:10.592 ***** 2021-11-20T05:54:10.844 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-11-20T05:54:10.846 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:10.867 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:10 +0000 (0:00:00.050) 0:05:10.642 ***** 2021-11-20T05:54:10.915 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-11-20T05:54:10.917 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:54:11.062 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-11-20T05:54:11.064 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:54:11.085 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:11 +0000 (0:00:00.217) 0:05:10.860 ***** 2021-11-20T05:54:11.093 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-11-20T05:54:11.096 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-20T05:54:11.097 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:11.123 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:11 +0000 (0:00:00.038) 0:05:10.898 ***** 2021-11-20T05:54:11.130 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-11-20T05:54:11.132 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-11-20T05:54:11.157 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:11 +0000 (0:00:00.034) 0:05:10.933 ***** 2021-11-20T05:54:11.164 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-11-20T05:54:11.166 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:11.189 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:11 +0000 (0:00:00.032) 0:05:10.965 ***** 2021-11-20T05:54:11.217 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-11-20T05:54:11.219 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:11.241 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:11 +0000 (0:00:00.051) 0:05:11.017 ***** 2021-11-20T05:54:11.437 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-11-20T05:54:11.439 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:54:11.459 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:11 +0000 (0:00:00.217) 0:05:11.235 ***** 2021-11-20T05:54:11.496 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-11-20T05:54:11.498 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-11-20T05:54:11.518 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:11 +0000 (0:00:00.059) 0:05:11.294 ***** 2021-11-20T05:54:11.526 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-11-20T05:54:11.528 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true} 2021-11-20T05:54:11.553 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:11 +0000 (0:00:00.034) 0:05:11.328 ***** 2021-11-20T05:54:11.737 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true} 2021-11-20T05:54:11.763 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:11 +0000 (0:00:00.209) 0:05:11.538 ***** 2021-11-20T05:54:12.124 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-11-20T05:54:12.126 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-11-20T05:54:12.148 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.384) 0:05:11.923 ***** 2021-11-20T05:54:12.170 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.022) 0:05:11.945 ***** 2021-11-20T05:54:12.187 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-11-20T05:54:12.189 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.214 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.044) 0:05:11.989 ***** 2021-11-20T05:54:12.225 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.255 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.040) 0:05:12.030 ***** 2021-11-20T05:54:12.261 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-11-20T05:54:12.263 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.286 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.030) 0:05:12.061 ***** 2021-11-20T05:54:12.292 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.315 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.029) 0:05:12.091 ***** 2021-11-20T05:54:12.321 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-11-20T05:54:12.323 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.345 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.029) 0:05:12.121 ***** 2021-11-20T05:54:12.352 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.376 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.031) 0:05:12.152 ***** 2021-11-20T05:54:12.382 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-11-20T05:54:12.384 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.406 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.029) 0:05:12.182 ***** 2021-11-20T05:54:12.413 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.436 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.030) 0:05:12.212 ***** 2021-11-20T05:54:12.442 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-11-20T05:54:12.444 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.470 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.033) 0:05:12.246 ***** 2021-11-20T05:54:12.478 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.501 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.031) 0:05:12.277 ***** 2021-11-20T05:54:12.507 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-11-20T05:54:12.509 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.531 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.030) 0:05:12.307 ***** 2021-11-20T05:54:12.538 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.561 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.029) 0:05:12.337 ***** 2021-11-20T05:54:12.567 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-11-20T05:54:12.569 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.592 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.030) 0:05:12.367 ***** 2021-11-20T05:54:12.598 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.621 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.029) 0:05:12.397 ***** 2021-11-20T05:54:12.627 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-11-20T05:54:12.629 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.652 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.030) 0:05:12.427 ***** 2021-11-20T05:54:12.658 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.681 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.029) 0:05:12.457 ***** 2021-11-20T05:54:12.688 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-11-20T05:54:12.689 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.712 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.030) 0:05:12.488 ***** 2021-11-20T05:54:12.719 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.745 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.032) 0:05:12.520 ***** 2021-11-20T05:54:12.750 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-11-20T05:54:12.752 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.774 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.029) 0:05:12.550 ***** 2021-11-20T05:54:12.782 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.804 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.029) 0:05:12.580 ***** 2021-11-20T05:54:12.811 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-11-20T05:54:12.812 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.836 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.031) 0:05:12.611 ***** 2021-11-20T05:54:12.843 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.867 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.030) 0:05:12.642 ***** 2021-11-20T05:54:12.872 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-11-20T05:54:12.874 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.897 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.030) 0:05:12.673 ***** 2021-11-20T05:54:12.905 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.928 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.030) 0:05:12.703 ***** 2021-11-20T05:54:12.933 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-11-20T05:54:12.935 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:12.967 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:12 +0000 (0:00:00.038) 0:05:12.742 ***** 2021-11-20T05:54:12.977 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:13.010 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:13 +0000 (0:00:00.043) 0:05:12.786 ***** 2021-11-20T05:54:13.018 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-11-20T05:54:13.020 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:13.042 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:13 +0000 (0:00:00.031) 0:05:12.818 ***** 2021-11-20T05:54:13.050 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:13.073 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:13 +0000 (0:00:00.030) 0:05:12.849 ***** 2021-11-20T05:54:13.079 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-11-20T05:54:13.081 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:13.103 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:13 +0000 (0:00:00.029) 0:05:12.878 ***** 2021-11-20T05:54:13.109 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:13.132 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:13 +0000 (0:00:00.029) 0:05:12.908 ***** 2021-11-20T05:54:13.573 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-11-20T05:54:13.575 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-11-20T05:54:13.597 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:13 +0000 (0:00:00.464) 0:05:13.372 ***** 2021-11-20T05:54:13.605 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-11-20T05:54:13.629 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:13 +0000 (0:00:00.032) 0:05:13.404 ***** 2021-11-20T05:54:13.896 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-11-20T05:54:13.897 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 05:48:07 UTC", "ActiveEnterTimestampMonotonic": "12767212", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service sysinit.target basic.target system.slice -.mount ntpd.service sntp.service ntpdate.service systemd-journald.socket tmp.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 05:48:06 UTC", "AssertTimestampMonotonic": "11550173", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 05:48:06 UTC", "ConditionTimestampMonotonic": "11550141", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd 2021-11-20T05:54:13.899 INFO:teuthology.task.ansible.out:.service systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "886", "ExecMainStartTimestamp": "Sat 2021-11-20 05:48:07 UTC", "ExecMainStartTimestampMonotonic": "12705895", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 05:48:06 UTC", "InactiveExitTimestampMonotonic": "11589540", "InvocationID": "673f669298424717ae829e5dfbebac1b", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": 2021-11-20T05:54:13.901 INFO:teuthology.task.ansible.out:"infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "886", "MemoryAccounting": "yes", "MemoryCurrent": "1687552", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice 2021-11-20T05:54:13.902 INFO:teuthology.task.ansible.out:-.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 05:48:07 UTC", "StateChangeTimestampMonotonic": "12767212", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 05:48:07 UTC", "WatchdogTimestampMonotonic": "12705913", "WatchdogUSec": "0"}} 2021-11-20T05:54:13.922 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:13 +0000 (0:00:00.292) 0:05:13.696 ***** 2021-11-20T05:54:13.932 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-11-20 05:48:21 UTC", "ActiveEnterTimestampMonotonic": "12775495", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "ntpdate.service systemd-tmpfiles-setup.service sntp.service systemd-journald.socket tmp.mount ntpd.service basic.target -.mount sysinit.target system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-11-20 05:48:19 UTC", "AssertTimestampMonotonic": "11228556", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-11-20 05:48:19 UTC", "ConditionTimestampMonotonic": "11228519", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd 2021-11-20T05:54:13.933 INFO:teuthology.task.ansible.out:.service shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "891", "ExecMainStartTimestamp": "Sat 2021-11-20 05:48:21 UTC", "ExecMainStartTimestampMonotonic": "12685818", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-11-20 05:48:19 UTC", "InactiveExitTimestampMonotonic": "11239040", "InvocationID": "dcfcf00eb45940139823242b3de3712a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": 2021-11-20T05:54:13.935 INFO:teuthology.task.ansible.out:"infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127242", "LimitNPROCSoft": "127242", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127242", "LimitSIGPENDINGSoft": "127242", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "891", "MemoryAccounting": "yes", "MemoryCurrent": "1601536", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit 2021-11-20T05:54:13.937 INFO:teuthology.task.ansible.out:.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-11-20 05:48:21 UTC", "StateChangeTimestampMonotonic": "12775495", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203587", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sat 2021-11-20 05:48:21 UTC", "WatchdogTimestampMonotonic": "12685835", "WatchdogUSec": "0"}} 2021-11-20T05:54:13.955 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:13 +0000 (0:00:00.033) 0:05:13.730 ***** 2021-11-20T05:54:13.962 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:54:13.963 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-11-20T05:54:13.988 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:13 +0000 (0:00:00.032) 0:05:13.763 ***** 2021-11-20T05:54:13.994 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-11-20T05:54:14.020 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:14 +0000 (0:00:00.033) 0:05:13.796 ***** 2021-11-20T05:54:14.028 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:54:14.029 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-11-20T05:54:14.054 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:14 +0000 (0:00:00.033) 0:05:13.829 ***** 2021-11-20T05:54:14.061 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-11-20T05:54:14.085 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:14 +0000 (0:00:00.031) 0:05:13.860 ***** 2021-11-20T05:54:14.098 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:14.115 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:14 +0000 (0:00:00.029) 0:05:13.890 ***** 2021-11-20T05:54:14.122 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:14.146 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:14 +0000 (0:00:00.030) 0:05:13.921 ***** 2021-11-20T05:54:14.153 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-11-20T05:54:14.155 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:14.176 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:14 +0000 (0:00:00.030) 0:05:13.952 ***** 2021-11-20T05:54:14.183 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:14.206 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:14 +0000 (0:00:00.030) 0:05:13.982 ***** 2021-11-20T05:54:14.213 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-11-20T05:54:14.214 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:14.235 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:14 +0000 (0:00:00.028) 0:05:14.011 ***** 2021-11-20T05:54:14.242 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:14.272 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:14 +0000 (0:00:00.036) 0:05:14.048 ***** 2021-11-20T05:54:14.434 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-11-20T05:54:14.436 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-11-20T05:54:14.458 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:14 +0000 (0:00:00.185) 0:05:14.234 ***** 2021-11-20T05:54:14.484 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-11-20T05:54:14.507 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:14 +0000 (0:00:00.049) 0:05:14.283 ***** 2021-11-20T05:54:14.929 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-11-20T05:54:14.953 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:14 +0000 (0:00:00.445) 0:05:14.728 ***** 2021-11-20T05:54:14.971 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-11-20T05:54:14.995 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:14 +0000 (0:00:00.042) 0:05:14.771 ***** 2021-11-20T05:54:16.051 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-11-20T05:54:16.052 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-11-20T05:54:16.073 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:16 +0000 (0:00:01.077) 0:05:15.849 ***** 2021-11-20T05:54:16.091 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-11-20T05:54:16.114 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:16 +0000 (0:00:00.040) 0:05:15.889 ***** 2021-11-20T05:54:16.121 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-11-20T05:54:16.123 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:16.145 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:16 +0000 (0:00:00.031) 0:05:15.920 ***** 2021-11-20T05:54:16.152 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:16.179 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:16 +0000 (0:00:00.033) 0:05:15.954 ***** 2021-11-20T05:54:16.186 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-11-20T05:54:16.188 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-11-20T05:54:16.214 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:16 +0000 (0:00:00.035) 0:05:15.990 ***** 2021-11-20T05:54:16.221 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-11-20T05:54:16.245 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:16 +0000 (0:00:00.031) 0:05:16.021 ***** 2021-11-20T05:54:16.400 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-11-20T05:54:16.427 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:16 +0000 (0:00:00.181) 0:05:16.203 ***** 2021-11-20T05:54:16.436 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-11-20T05:54:16.761 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:16 +0000 (0:00:00.038) 0:05:16.241 ***** TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Saturday 20 November 2021 05:54:16 +0000 (0:00:00.041) 0:05:16.283 ***** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Saturday 20 November 2021 05:54:16 +0000 (0:00:00.035) 0:05:16.319 ***** TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} Saturday 20 November 2021 05:54:16 +0000 (0:00:00.033) 0:05:16.352 ***** ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} Saturday 20 November 2021 05:54:16 +0000 (0:00:00.030) 0:05:16.383 ***** TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Saturday 20 November 2021 05:54:16 +0000 (0:00:00.030) 0:05:16.413 ***** skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Saturday 20 November 2021 05:54:16 +0000 (0:00:00.030) 0:05:16.444 ***** 2021-11-20T05:54:16.831 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-11-20T05:54:16.833 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-11-20T05:54:16.855 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:16 +0000 (0:00:00.186) 0:05:16.630 ***** 2021-11-20T05:54:16.862 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-11-20T05:54:16.889 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:16 +0000 (0:00:00.033) 0:05:16.664 ***** 2021-11-20T05:54:17.043 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-11-20T05:54:17.045 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-11-20T05:54:17.070 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.180) 0:05:16.845 ***** 2021-11-20T05:54:17.092 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-11-20T05:54:17.115 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.045) 0:05:16.891 ***** 2021-11-20T05:54:17.123 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.146 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.031) 0:05:16.922 ***** 2021-11-20T05:54:17.154 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.178 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.031) 0:05:16.954 ***** 2021-11-20T05:54:17.187 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-11-20T05:54:17.189 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.189 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.190 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.201 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.204 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.228 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.049) 0:05:17.003 ***** 2021-11-20T05:54:17.236 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.238 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.238 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.239 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.264 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.036) 0:05:17.040 ***** 2021-11-20T05:54:17.283 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-11-20T05:54:17.285 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi178.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.286 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditiona 2021-11-20T05:54:17.288 INFO:teuthology.task.ansible.out:l result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.310 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.045) 0:05:17.086 ***** 2021-11-20T05:54:17.319 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.324 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi139.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.348 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.037) 0:05:17.123 ***** 2021-11-20T05:54:17.355 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-20T05:54:17.356 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.382 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.033) 0:05:17.157 ***** 2021-11-20T05:54:17.388 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.415 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.032) 0:05:17.190 ***** 2021-11-20T05:54:17.423 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-11-20T05:54:17.425 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.452 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.037) 0:05:17.227 ***** 2021-11-20T05:54:17.459 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.484 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.032) 0:05:17.260 ***** 2021-11-20T05:54:17.492 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-11-20T05:54:17.493 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.520 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.035) 0:05:17.295 ***** 2021-11-20T05:54:17.529 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.557 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.036) 0:05:17.332 ***** 2021-11-20T05:54:17.563 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-11-20T05:54:17.566 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.596 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.039) 0:05:17.371 ***** 2021-11-20T05:54:17.603 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.628 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.032) 0:05:17.404 ***** 2021-11-20T05:54:17.636 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-11-20T05:54:17.638 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.667 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.037) 0:05:17.441 ***** 2021-11-20T05:54:17.677 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.706 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.039) 0:05:17.481 ***** 2021-11-20T05:54:17.713 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-11-20T05:54:17.715 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.742 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.035) 0:05:17.517 ***** 2021-11-20T05:54:17.754 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.782 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.040) 0:05:17.557 ***** 2021-11-20T05:54:17.791 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-11-20T05:54:17.793 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.823 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.040) 0:05:17.598 ***** 2021-11-20T05:54:17.831 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.856 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.033) 0:05:17.632 ***** 2021-11-20T05:54:17.864 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-11-20T05:54:17.866 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.888 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.031) 0:05:17.664 ***** 2021-11-20T05:54:17.896 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:17.920 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:17 +0000 (0:00:00.031) 0:05:17.695 ***** 2021-11-20T05:54:18.105 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-11-20T05:54:18.106 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005174", "end": "2021-11-20 05:54:18.089613", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-11-20 05:54:18.084439", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:18.139 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005130", "end": "2021-11-20 05:54:18.117490", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-11-20 05:54:18.112360", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:18.292 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003142", "end": "2021-11-20 05:54:18.264406", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-11-20 05:54:18.261264", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:18.315 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003054", "end": "2021-11-20 05:54:18.295230", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-11-20 05:54:18.292176", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:18.452 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002867", "end": "2021-11-20 05:54:18.435138", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 05:54:18.432271", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:18.484 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002834", "end": "2021-11-20 05:54:18.466621", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-11-20 05:54:18.463787", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:18.621 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003304", "end": "2021-11-20 05:54:18.605818", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-11-20 05:54:18.602514", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:18.647 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:18 +0000 (0:00:00.727) 0:05:18.423 ***** 2021-11-20T05:54:18.649 INFO:teuthology.task.ansible.out: 2021-11-20T05:54:18.655 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003115", "end": "2021-11-20 05:54:18.636586", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-11-20 05:54:18.633471", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:54:18.681 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:18 +0000 (0:00:00.033) 0:05:18.456 ***** 2021-11-20T05:54:18.688 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-11-20T05:54:18.713 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:18 +0000 (0:00:00.032) 0:05:18.489 ***** 2021-11-20T05:54:18.749 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:18 +0000 (0:00:00.035) 0:05:18.524 ***** 2021-11-20T05:54:18.756 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-11-20T05:54:18.759 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:18.781 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:18.858 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-11-20T05:54:18.875 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:18 +0000 (0:00:00.126) 0:05:18.650 ***** 2021-11-20T05:54:18.895 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:18 +0000 (0:00:00.020) 0:05:18.671 ***** 2021-11-20T05:54:20.535 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi139.front.sepia.ceph.com] 2021-11-20T05:54:20.567 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] 2021-11-20T05:54:20.593 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:20 +0000 (0:00:01.698) 0:05:20.369 ***** 2021-11-20T05:54:20.623 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T05:54:20.625 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:20.649 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:20 +0000 (0:00:00.055) 0:05:20.425 ***** 2021-11-20T05:54:20.671 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:20 +0000 (0:00:00.019) 0:05:20.444 ***** 2021-11-20T05:54:20.688 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-11-20T05:54:20.711 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:20 +0000 (0:00:00.041) 0:05:20.486 ***** 2021-11-20T05:54:20.718 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T05:54:20.720 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:54:20.743 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:20 +0000 (0:00:00.032) 0:05:20.519 ***** 2021-11-20T05:54:20.760 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-11-20T05:54:20.762 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-11-20T05:54:20.786 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:20 +0000 (0:00:00.042) 0:05:20.561 ***** 2021-11-20T05:54:20.793 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T05:54:20.794 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-11-20T05:54:20.817 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:20 +0000 (0:00:00.031) 0:05:20.592 ***** 2021-11-20T05:54:20.864 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-11-20T05:54:20.866 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-11-20T05:54:20.888 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:54:20 +0000 (0:00:00.070) 0:05:20.663 ***** 2021-11-20T05:55:02.845 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-11-20T05:55:02.849 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: containers-common-2:1-6.module_el8.6.0+954+963caf36.noarch", "Installed: podman-catatonit-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Installed: container-selinux-2:2.170.0-1.module_el8.6.0+954+963caf36.noarch", "Installed: podman-docker-1:3.4.1-3.module_el8.6.0+954+963caf36.noarch", "Installed: podman-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Removed: container-selinux-2:2.159.0-1.module_el8.5.0+733+9bb5dffa.noarch", "Removed: containers-common-1:1.2.2-4.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-catatonit-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-docker-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.noarch"]} 2021-11-20T05:55:02.875 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:02 +0000 (0:00:41.987) 0:06:02.650 ***** 2021-11-20T05:55:02.914 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi139.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-11-20T05:55:02.938 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:02 +0000 (0:00:00.063) 0:06:02.714 ***** 2021-11-20T05:55:02.979 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T05:55:02.981 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:55:03.004 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:03 +0000 (0:00:00.065) 0:06:02.779 ***** 2021-11-20T05:55:03.236 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-11-20T05:55:03.237 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:55:03.427 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:55:03.453 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:03 +0000 (0:00:00.448) 0:06:03.228 ***** 2021-11-20T05:55:03.512 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-11-20T05:55:03.514 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: containers-common-2:1-6.module_el8.6.0+954+963caf36.noarch", "Installed: podman-catatonit-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Installed: container-selinux-2:2.170.0-1.module_el8.6.0+954+963caf36.noarch", "Installed: podman-docker-1:3.4.1-3.module_el8.6.0+954+963caf36.noarch", "Installed: podman-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Removed: container-selinux-2:2.159.0-1.module_el8.5.0+733+9bb5dffa.noarch", "Removed: containers-common-1:1.2.2-4.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-catatonit-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-docker-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.noarch"]} 2021-11-20T05:55:03.536 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:03 +0000 (0:00:00.083) 0:06:03.312 ***** 2021-11-20T05:55:03.564 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T05:55:03.566 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-11-20T05:55:03.589 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:03 +0000 (0:00:00.052) 0:06:03.364 ***** 2021-11-20T05:55:03.627 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-20T05:55:03.630 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:55:03.650 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:03 +0000 (0:00:00.061) 0:06:03.426 ***** 2021-11-20T05:55:03.881 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-11-20T05:55:03.882 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:55:03.987 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-11-20T05:55:03.989 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387703.5314806-8826-5530843603769/source", "state": "file", "uid": 0} 2021-11-20T05:55:04.062 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-11-20T05:55:04.064 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-20T05:55:04.090 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:04 +0000 (0:00:00.439) 0:06:03.865 ***** 2021-11-20T05:55:04.477 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-11-20T05:55:04.479 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387704.0279758-8826-165753427376633/source", "state": "file", "uid": 0} 2021-11-20T05:55:04.503 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:04 +0000 (0:00:00.413) 0:06:04.279 ***** 2021-11-20T05:55:04.629 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387704.1582925-8903-177615159613164/source", "state": "file", "uid": 0} 2021-11-20T05:55:05.123 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637387704.6645212-8903-224004163705985/source", "state": "file", "uid": 0} 2021-11-20T05:55:05.152 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:05 +0000 (0:00:00.648) 0:06:04.927 ***** 2021-11-20T05:55:06.513 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** 2021-11-20T05:55:06.515 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:55:06.537 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:06 +0000 (0:00:01.384) 0:06:06.312 ***** 2021-11-20T05:55:07.164 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-20T05:55:07.189 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:07 +0000 (0:00:00.652) 0:06:06.965 ***** 2021-11-20T05:55:09.674 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-11-20T05:55:09.676 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-esywx6w0-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-esywx6w0-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-11-20T05:55:09.677 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-11-20T05:55:09.698 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:09 +0000 (0:00:02.508) 0:06:09.473 ***** 2021-11-20T05:55:09.737 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-11-20T05:55:09.739 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:55:09.762 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:09 +0000 (0:00:00.063) 0:06:09.537 ***** 2021-11-20T05:55:09.799 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-11-20T05:55:09.801 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:55:09.824 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:09 +0000 (0:00:00.063) 0:06:09.600 ***** 2021-11-20T05:55:10.176 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-11-20T05:55:10.178 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.143263", "end": "2021-11-20 05:55:10.165739", "rc": 0, "start": "2021-11-20 05:55:10.022476", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:55:10.200 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:10 +0000 (0:00:00.375) 0:06:09.975 ***** 2021-11-20T05:55:10.248 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-11-20T05:55:10.250 INFO:teuthology.task.ansible.out:skipping: [smithi139.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:55:10.379 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-11-20T05:55:10.380 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-6650mauq-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-6650mauq-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-11-20T05:55:10.382 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-11-20T05:55:10.400 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:10 +0000 (0:00:00.201) 0:06:10.176 ***** 2021-11-20T05:55:10.429 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-11-20T05:55:10.431 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:55:10.451 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:10 +0000 (0:00:00.051) 0:06:10.227 ***** 2021-11-20T05:55:10.479 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-11-20T05:55:10.482 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:55:10.503 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:10 +0000 (0:00:00.051) 0:06:10.279 ***** 2021-11-20T05:55:10.854 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-11-20T05:55:10.856 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.151374", "end": "2021-11-20 05:55:10.843630", "rc": 0, "start": "2021-11-20 05:55:10.692256", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-20T05:55:10.876 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:10 +0000 (0:00:00.372) 0:06:10.651 ***** 2021-11-20T05:55:10.913 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-11-20T05:55:10.916 INFO:teuthology.task.ansible.out:skipping: [smithi178.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-20T05:55:10.982 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-11-20T05:55:10.984 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-11-20T05:55:10.986 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-11-20T05:55:10.998 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-11-20T05:55:10.999 INFO:teuthology.task.ansible.out:Saturday 20 November 2021 05:55:10 +0000 (0:00:00.122) 0:06:10.773 ***** 2021-11-20T05:55:11.643 INFO:teuthology.task.ansible.out:ok: [smithi139.front.sepia.ceph.com] 2021-11-20T05:55:12.690 INFO:teuthology.task.ansible.out:ok: [smithi178.front.sepia.ceph.com] 2021-11-20T05:55:12.716 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** Saturday 20 November 2021 05:55:12 +0000 (0:00:01.718) 0:06:12.492 ***** 2021-11-20T05:55:12.929 INFO:teuthology.task.ansible.out:changed: [smithi139.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T05:55:12.941 INFO:teuthology.task.ansible.out:changed: [smithi178.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-20T05:55:12.949 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* 2021-11-20T05:55:12.951 INFO:teuthology.task.ansible.out:smithi139.front.sepia.ceph.com : ok=138 changed=51 unreachable=0 failed=0 skipped=181 rescued=0 ignored=3 smithi178.front.sepia.ceph.com : ok=138 changed=51 unreachable=0 failed=0 skipped=181 rescued=0 ignored=3 Saturday 20 November 2021 05:55:12 +0000 (0:00:00.232) 0:06:12.725 ***** =============================================================================== Create all admin users with sudo access. ------------------------------- 59.43s container-host : Install container packages ---------------------------- 41.99s common : nrpe - Ensure SELinux policy is up to date -------------------- 38.39s users : Update authorized_keys using the keys repo --------------------- 32.02s Remove revoked users --------------------------------------------------- 18.93s testnode : Ensure ceph packages are not present. ----------------------- 18.16s common : nrpe - Remove SELinux policy package -------------------------- 15.90s common : nrpe - Load SELinux policy package ---------------------------- 15.54s testnode : Install packages --------------------------------------------- 8.86s 2021-11-20T05:55:12.952 INFO:teuthology.task.ansible.out:testnode : Zap all non-root disks --------------------------------------- 7.44s testnode : Upgrade packages --------------------------------------------- 7.44s testnode : Install GPG -------------------------------------------------- 7.19s common : Install Kerberos Packages (RedHat) ----------------------------- 4.44s testnode : Ensure ceph dependency packages are not present. ------------- 3.05s testnode : Install GPG keys --------------------------------------------- 2.56s testnode : Rebuild rpmdb ------------------------------------------------ 2.56s container-host : Install registries-conf-ctl ---------------------------- 2.51s testnode : Remove packages ---------------------------------------------- 1.77s common : Make sure smartmontools is installed --------------------------- 1.73s users : Update authorized_keys for each user with 2021-11-20T05:55:12.954 INFO:teuthology.task.ansible.out: literal keys ---------- 1.72s 2021-11-20T05:55:13.354 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi139.front.sepia.ceph.com'), Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2021-11-20T05:55:13.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2021-11-20T05:55:13.357 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:55:13.429 DEBUG:teuthology.orchestra.run.smithi139:> true 2021-11-20T05:55:13.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2021-11-20T05:55:13.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-20T05:55:13.568 DEBUG:teuthology.orchestra.run.smithi178:> true 2021-11-20T05:55:13.636 INFO:teuthology.run_tasks:Running task clock... 2021-11-20T05:55:13.647 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-11-20T05:55:13.648 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-11-20T05:55:13.650 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T05:55:13.653 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-11-20T05:55:13.653 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T05:55:13.680 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-11-20T05:55:13.694 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-11-20T05:55:13.708 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-11-20T05:55:13.723 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-11-20T05:55:13.728 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: ntpd: command not found 2021-11-20T05:55:13.738 INFO:teuthology.orchestra.run.smithi139.stdout:506 Cannot talk to daemon 2021-11-20T05:55:13.752 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-11-20T05:55:13.758 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: ntpd: command not found 2021-11-20T05:55:13.765 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-11-20T05:55:13.768 INFO:teuthology.orchestra.run.smithi178.stdout:506 Cannot talk to daemon 2021-11-20T05:55:13.782 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-11-20T05:55:13.795 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-11-20T05:55:13.840 INFO:teuthology.orchestra.run.smithi139.stderr:bash: ntpq: command not found 2021-11-20T05:55:13.843 INFO:teuthology.orchestra.run.smithi139.stdout:210 Number of sources = 4 2021-11-20T05:55:13.843 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T05:55:13.843 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2021-11-20T05:55:13.843 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:55:13.844 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:55:13.844 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:55:13.844 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:55:13.859 INFO:teuthology.orchestra.run.smithi178.stderr:bash: ntpq: command not found 2021-11-20T05:55:13.862 INFO:teuthology.orchestra.run.smithi178.stdout:210 Number of sources = 4 2021-11-20T05:55:13.862 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T05:55:13.862 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2021-11-20T05:55:13.862 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:55:13.863 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:55:13.863 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:55:13.863 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-20T05:55:13.863 INFO:teuthology.run_tasks:Running task pexec... 2021-11-20T05:55:13.878 INFO:teuthology.task.pexec:Executing custom commands... 2021-11-20T05:55:13.880 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi139.front.sepia.ceph.com 2021-11-20T05:55:13.880 DEBUG:teuthology.orchestra.run.smithi139:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-11-20T05:55:13.881 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi178.front.sepia.ceph.com 2021-11-20T05:55:13.882 DEBUG:teuthology.orchestra.run.smithi178:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-11-20T05:55:14.451 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:53 ago on Sat 20 Nov 2021 05:53:21 AM UTC. 2021-11-20T05:55:14.467 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:53 ago on Sat 20 Nov 2021 05:53:21 AM UTC. 2021-11-20T05:55:15.870 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:55:15.871 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:55:15.872 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2021-11-20T05:55:15.872 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:55:15.872 INFO:teuthology.orchestra.run.smithi139.stdout:Resetting modules: 2021-11-20T05:55:15.872 INFO:teuthology.orchestra.run.smithi139.stdout: container-tools 2021-11-20T05:55:15.872 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:55:15.873 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:55:15.873 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:55:15.873 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:55:15.915 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:55:15.917 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:55:15.917 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2021-11-20T05:55:15.918 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:55:15.918 INFO:teuthology.orchestra.run.smithi178.stdout:Resetting modules: 2021-11-20T05:55:15.918 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools 2021-11-20T05:55:15.919 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:55:15.919 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:55:15.920 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:55:15.920 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:55:16.093 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:55:16.157 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:55:16.693 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:55 ago on Sat 20 Nov 2021 05:53:21 AM UTC. 2021-11-20T05:55:16.764 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:55 ago on Sat 20 Nov 2021 05:53:21 AM UTC. 2021-11-20T05:55:18.104 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:55:18.108 INFO:teuthology.orchestra.run.smithi139.stdout:===================================================================================================== 2021-11-20T05:55:18.108 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-11-20T05:55:18.108 INFO:teuthology.orchestra.run.smithi139.stdout:===================================================================================================== 2021-11-20T05:55:18.109 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrading: 2021-11-20T05:55:18.109 INFO:teuthology.orchestra.run.smithi139.stdout: conmon x86_64 2:2.0.30-1.module_el8.6.0+944+d413f95e CentOS-AppStream 54 k 2021-11-20T05:55:18.109 INFO:teuthology.orchestra.run.smithi139.stdout: containernetworking-plugins x86_64 1.0.1-1.module_el8.6.0+944+d413f95e CentOS-AppStream 19 M 2021-11-20T05:55:18.109 INFO:teuthology.orchestra.run.smithi139.stdout: criu x86_64 3.15-3.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 518 k 2021-11-20T05:55:18.110 INFO:teuthology.orchestra.run.smithi139.stdout: fuse-overlayfs x86_64 1.7.1-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 73 k 2021-11-20T05:55:18.110 INFO:teuthology.orchestra.run.smithi139.stdout: libslirp x86_64 4.4.0-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 70 k 2021-11-20T05:55:18.110 INFO:teuthology.orchestra.run.smithi139.stdout: runc x86_64 1.0.2-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 3.1 M 2021-11-20T05:55:18.110 INFO:teuthology.orchestra.run.smithi139.stdout: slirp4netns x86_64 1.1.8-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 51 k 2021-11-20T05:55:18.111 INFO:teuthology.orchestra.run.smithi139.stdout:Installing group/module packages: 2021-11-20T05:55:18.111 INFO:teuthology.orchestra.run.smithi139.stdout: buildah x86_64 1:1.23.1-2.module_el8.6.0+954+963caf36 CentOS-AppStream 7.9 M 2021-11-20T05:55:18.111 INFO:teuthology.orchestra.run.smithi139.stdout: cockpit-podman noarch 35-1.module_el8.6.0+944+d413f95e CentOS-AppStream 457 k 2021-11-20T05:55:18.111 INFO:teuthology.orchestra.run.smithi139.stdout: crun x86_64 1.2-1.module_el8.6.0+954+963caf36 CentOS-AppStream 193 k 2021-11-20T05:55:18.112 INFO:teuthology.orchestra.run.smithi139.stdout: python3-podman noarch 3.2.1-4.module_el8.6.0+954+963caf36 CentOS-AppStream 148 k 2021-11-20T05:55:18.113 INFO:teuthology.orchestra.run.smithi139.stdout: skopeo x86_64 2:1.5.0-2.module_el8.6.0+954+963caf36 CentOS-AppStream 6.7 M 2021-11-20T05:55:18.113 INFO:teuthology.orchestra.run.smithi139.stdout: toolbox x86_64 0.0.99.3-1.module_el8.5.0+877+1c30e0c9 CentOS-AppStream 2.3 M 2021-11-20T05:55:18.113 INFO:teuthology.orchestra.run.smithi139.stdout: udica noarch 0.2.6-2.module_el8.6.0+944+d413f95e CentOS-AppStream 48 k 2021-11-20T05:55:18.113 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-11-20T05:55:18.113 INFO:teuthology.orchestra.run.smithi139.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-11-20T05:55:18.114 INFO:teuthology.orchestra.run.smithi139.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-11-20T05:55:18.114 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-11-20T05:55:18.114 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 CentOS-AppStream 25 k 2021-11-20T05:55:18.114 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyxdg noarch 0.25-16.el8 CentOS-AppStream 94 k 2021-11-20T05:55:18.115 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-11-20T05:55:18.115 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-11-20T05:55:18.115 INFO:teuthology.orchestra.run.smithi139.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-11-20T05:55:18.115 INFO:teuthology.orchestra.run.smithi139.stdout:Installing module profiles: 2021-11-20T05:55:18.116 INFO:teuthology.orchestra.run.smithi139.stdout: container-tools/common 2021-11-20T05:55:18.116 INFO:teuthology.orchestra.run.smithi139.stdout:Enabling module streams: 2021-11-20T05:55:18.116 INFO:teuthology.orchestra.run.smithi139.stdout: container-tools rhel8 2021-11-20T05:55:18.117 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:55:18.117 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:55:18.117 INFO:teuthology.orchestra.run.smithi139.stdout:===================================================================================================== 2021-11-20T05:55:18.117 INFO:teuthology.orchestra.run.smithi139.stdout:Install 15 Packages 2021-11-20T05:55:18.118 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrade 7 Packages 2021-11-20T05:55:18.118 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:55:18.119 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 41 M 2021-11-20T05:55:18.119 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:55:18.165 INFO:teuthology.orchestra.run.smithi139.stdout:(1/22): crun-1.2-1.module_el8.6.0+954+963caf36. 3.7 MB/s | 193 kB 00:00 2021-11-20T05:55:18.165 INFO:teuthology.orchestra.run.smithi139.stdout:(2/22): cockpit-podman-35-1.module_el8.6.0+944+ 8.5 MB/s | 457 kB 00:00 2021-11-20T05:55:18.166 INFO:teuthology.orchestra.run.smithi139.stdout:(3/22): python3-pytoml-0.1.14-5.git7dea353.el8. 14 MB/s | 25 kB 00:00 2021-11-20T05:55:18.168 INFO:teuthology.orchestra.run.smithi139.stdout:(4/22): python3-podman-3.2.1-4.module_el8.6.0+9 28 MB/s | 148 kB 00:00 2021-11-20T05:55:18.170 INFO:teuthology.orchestra.run.smithi139.stdout:(5/22): python3-pyxdg-0.25-16.el8.noarch.rpm 25 MB/s | 94 kB 00:00 2021-11-20T05:55:18.255 INFO:teuthology.orchestra.run.smithi139.stdout:(6/22): toolbox-0.0.99.3-1.module_el8.5.0+877+1 27 MB/s | 2.3 MB 00:00 2021-11-20T05:55:18.258 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:55:18.260 INFO:teuthology.orchestra.run.smithi139.stdout:(7/22): udica-0.2.6-2.module_el8.6.0+944+d413f9 9.7 MB/s | 48 kB 00:00 2021-11-20T05:55:18.261 INFO:teuthology.orchestra.run.smithi178.stdout:===================================================================================================== 2021-11-20T05:55:18.261 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2021-11-20T05:55:18.261 INFO:teuthology.orchestra.run.smithi178.stdout:===================================================================================================== 2021-11-20T05:55:18.262 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2021-11-20T05:55:18.262 INFO:teuthology.orchestra.run.smithi178.stdout: conmon x86_64 2:2.0.30-1.module_el8.6.0+944+d413f95e CentOS-AppStream 54 k 2021-11-20T05:55:18.262 INFO:teuthology.orchestra.run.smithi178.stdout: containernetworking-plugins x86_64 1.0.1-1.module_el8.6.0+944+d413f95e CentOS-AppStream 19 M 2021-11-20T05:55:18.262 INFO:teuthology.orchestra.run.smithi178.stdout: criu x86_64 3.15-3.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 518 k 2021-11-20T05:55:18.263 INFO:teuthology.orchestra.run.smithi178.stdout: fuse-overlayfs x86_64 1.7.1-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 73 k 2021-11-20T05:55:18.263 INFO:teuthology.orchestra.run.smithi178.stdout: libslirp x86_64 4.4.0-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 70 k 2021-11-20T05:55:18.263 INFO:teuthology.orchestra.run.smithi178.stdout: runc x86_64 1.0.2-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 3.1 M 2021-11-20T05:55:18.263 INFO:teuthology.orchestra.run.smithi178.stdout: slirp4netns x86_64 1.1.8-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 51 k 2021-11-20T05:55:18.263 INFO:teuthology.orchestra.run.smithi178.stdout:Installing group/module packages: 2021-11-20T05:55:18.264 INFO:teuthology.orchestra.run.smithi178.stdout: buildah x86_64 1:1.23.1-2.module_el8.6.0+954+963caf36 CentOS-AppStream 7.9 M 2021-11-20T05:55:18.264 INFO:teuthology.orchestra.run.smithi178.stdout: cockpit-podman noarch 35-1.module_el8.6.0+944+d413f95e CentOS-AppStream 457 k 2021-11-20T05:55:18.264 INFO:teuthology.orchestra.run.smithi178.stdout: crun x86_64 1.2-1.module_el8.6.0+954+963caf36 CentOS-AppStream 193 k 2021-11-20T05:55:18.264 INFO:teuthology.orchestra.run.smithi178.stdout: python3-podman noarch 3.2.1-4.module_el8.6.0+954+963caf36 CentOS-AppStream 148 k 2021-11-20T05:55:18.265 INFO:teuthology.orchestra.run.smithi178.stdout: skopeo x86_64 2:1.5.0-2.module_el8.6.0+954+963caf36 CentOS-AppStream 6.7 M 2021-11-20T05:55:18.265 INFO:teuthology.orchestra.run.smithi178.stdout: toolbox x86_64 0.0.99.3-1.module_el8.5.0+877+1c30e0c9 CentOS-AppStream 2.3 M 2021-11-20T05:55:18.265 INFO:teuthology.orchestra.run.smithi178.stdout: udica noarch 0.2.6-2.module_el8.6.0+944+d413f95e CentOS-AppStream 48 k 2021-11-20T05:55:18.265 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2021-11-20T05:55:18.266 INFO:teuthology.orchestra.run.smithi178.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-11-20T05:55:18.266 INFO:teuthology.orchestra.run.smithi178.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-11-20T05:55:18.266 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-11-20T05:55:18.266 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 CentOS-AppStream 25 k 2021-11-20T05:55:18.266 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyxdg noarch 0.25-16.el8 CentOS-AppStream 94 k 2021-11-20T05:55:18.267 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-11-20T05:55:18.267 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-11-20T05:55:18.267 INFO:teuthology.orchestra.run.smithi178.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-11-20T05:55:18.267 INFO:teuthology.orchestra.run.smithi178.stdout:Installing module profiles: 2021-11-20T05:55:18.268 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools/common 2021-11-20T05:55:18.268 INFO:teuthology.orchestra.run.smithi178.stdout:Enabling module streams: 2021-11-20T05:55:18.268 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools rhel8 2021-11-20T05:55:18.268 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:55:18.268 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:55:18.269 INFO:teuthology.orchestra.run.smithi178.stdout:===================================================================================================== 2021-11-20T05:55:18.269 INFO:teuthology.orchestra.run.smithi178.stdout:Install 15 Packages 2021-11-20T05:55:18.269 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 7 Packages 2021-11-20T05:55:18.269 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:55:18.270 INFO:teuthology.orchestra.run.smithi139.stdout:(8/22): yajl-2.1.0-10.el8.x86_64.rpm 8.4 MB/s | 41 kB 00:00 2021-11-20T05:55:18.270 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 41 M 2021-11-20T05:55:18.271 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:55:18.301 INFO:teuthology.orchestra.run.smithi139.stdout:(9/22): buildah-1.23.1-2.module_el8.6.0+954+963 44 MB/s | 7.9 MB 00:00 2021-11-20T05:55:18.301 INFO:teuthology.orchestra.run.smithi139.stdout:(10/22): python3-chardet-3.0.4-7.el8.noarch.rpm 6.7 MB/s | 195 kB 00:00 2021-11-20T05:55:18.302 INFO:teuthology.orchestra.run.smithi139.stdout:(11/22): python3-idna-2.5-5.el8.noarch.rpm 26 MB/s | 97 kB 00:00 2021-11-20T05:55:18.302 INFO:teuthology.orchestra.run.smithi139.stdout:(12/22): python3-pysocks-1.6.8-3.el8.noarch.rpm 11 MB/s | 34 kB 00:00 2021-11-20T05:55:18.302 INFO:teuthology.orchestra.run.smithi139.stdout:(13/22): python3-requests-2.20.0-2.1.el8_1.noar 27 MB/s | 123 kB 00:00 2021-11-20T05:55:18.308 INFO:teuthology.orchestra.run.smithi139.stdout:(14/22): python3-urllib3-1.24.2-5.el8.noarch.rp 25 MB/s | 177 kB 00:00 2021-11-20T05:55:18.308 INFO:teuthology.orchestra.run.smithi139.stdout:(15/22): conmon-2.0.30-1.module_el8.6.0+944+d41 13 MB/s | 54 kB 00:00 2021-11-20T05:55:18.319 INFO:teuthology.orchestra.run.smithi139.stdout:(16/22): criu-3.15-3.module_el8.6.0+926+8bef8ae 38 MB/s | 518 kB 00:00 2021-11-20T05:55:18.325 INFO:teuthology.orchestra.run.smithi139.stdout:(17/22): fuse-overlayfs-1.7.1-1.module_el8.6.0+ 13 MB/s | 73 kB 00:00 2021-11-20T05:55:18.330 INFO:teuthology.orchestra.run.smithi139.stdout:(18/22): libslirp-4.4.0-1.module_el8.6.0+926+8b 13 MB/s | 70 kB 00:00 2021-11-20T05:55:18.344 INFO:teuthology.orchestra.run.smithi178.stdout:(1/22): crun-1.2-1.module_el8.6.0+954+963caf36. 2.4 MB/s | 193 kB 00:00 2021-11-20T05:55:18.368 INFO:teuthology.orchestra.run.smithi178.stdout:(2/22): python3-podman-3.2.1-4.module_el8.6.0+9 6.1 MB/s | 148 kB 00:00 2021-11-20T05:55:18.377 INFO:teuthology.orchestra.run.smithi178.stdout:(3/22): python3-pytoml-0.1.14-5.git7dea353.el8. 2.8 MB/s | 25 kB 00:00 2021-11-20T05:55:18.394 INFO:teuthology.orchestra.run.smithi178.stdout:(4/22): cockpit-podman-35-1.module_el8.6.0+944+ 3.5 MB/s | 457 kB 00:00 2021-11-20T05:55:18.395 INFO:teuthology.orchestra.run.smithi178.stdout:(5/22): python3-pyxdg-0.25-16.el8.noarch.rpm 5.4 MB/s | 94 kB 00:00 2021-11-20T05:55:18.413 INFO:teuthology.orchestra.run.smithi139.stdout:(19/22): skopeo-1.5.0-2.module_el8.6.0+954+963c 28 MB/s | 6.7 MB 00:00 2021-11-20T05:55:18.422 INFO:teuthology.orchestra.run.smithi139.stdout:(20/22): runc-1.0.2-1.module_el8.6.0+926+8bef8a 34 MB/s | 3.1 MB 00:00 2021-11-20T05:55:18.423 INFO:teuthology.orchestra.run.smithi139.stdout:(21/22): slirp4netns-1.1.8-1.module_el8.6.0+926 5.0 MB/s | 51 kB 00:00 2021-11-20T05:55:18.528 INFO:teuthology.orchestra.run.smithi178.stdout:(6/22): buildah-1.23.1-2.module_el8.6.0+954+963 30 MB/s | 7.9 MB 00:00 2021-11-20T05:55:18.534 INFO:teuthology.orchestra.run.smithi178.stdout:(7/22): toolbox-0.0.99.3-1.module_el8.5.0+877+1 16 MB/s | 2.3 MB 00:00 2021-11-20T05:55:18.535 INFO:teuthology.orchestra.run.smithi178.stdout:(8/22): udica-0.2.6-2.module_el8.6.0+944+d413f9 6.5 MB/s | 48 kB 00:00 2021-11-20T05:55:18.537 INFO:teuthology.orchestra.run.smithi178.stdout:(9/22): yajl-2.1.0-10.el8.x86_64.rpm 13 MB/s | 41 kB 00:00 2021-11-20T05:55:18.541 INFO:teuthology.orchestra.run.smithi178.stdout:(10/22): python3-chardet-3.0.4-7.el8.noarch.rpm 35 MB/s | 195 kB 00:00 2021-11-20T05:55:18.595 INFO:teuthology.orchestra.run.smithi178.stdout:(11/22): python3-idna-2.5-5.el8.noarch.rpm 1.7 MB/s | 97 kB 00:00 2021-11-20T05:55:18.596 INFO:teuthology.orchestra.run.smithi178.stdout:(12/22): python3-pysocks-1.6.8-3.el8.noarch.rpm 638 kB/s | 34 kB 00:00 2021-11-20T05:55:18.601 INFO:teuthology.orchestra.run.smithi178.stdout:(13/22): python3-urllib3-1.24.2-5.el8.noarch.rp 30 MB/s | 177 kB 00:00 2021-11-20T05:55:18.602 INFO:teuthology.orchestra.run.smithi178.stdout:(14/22): python3-requests-2.20.0-2.1.el8_1.noar 15 MB/s | 123 kB 00:00 2021-11-20T05:55:18.605 INFO:teuthology.orchestra.run.smithi178.stdout:(15/22): conmon-2.0.30-1.module_el8.6.0+944+d41 15 MB/s | 54 kB 00:00 2021-11-20T05:55:18.620 INFO:teuthology.orchestra.run.smithi178.stdout:(16/22): criu-3.15-3.module_el8.6.0+926+8bef8ae 35 MB/s | 518 kB 00:00 2021-11-20T05:55:18.625 INFO:teuthology.orchestra.run.smithi178.stdout:(17/22): fuse-overlayfs-1.7.1-1.module_el8.6.0+ 13 MB/s | 73 kB 00:00 2021-11-20T05:55:18.630 INFO:teuthology.orchestra.run.smithi178.stdout:(18/22): libslirp-4.4.0-1.module_el8.6.0+926+8b 14 MB/s | 70 kB 00:00 2021-11-20T05:55:18.710 INFO:teuthology.orchestra.run.smithi178.stdout:(19/22): runc-1.0.2-1.module_el8.6.0+926+8bef8a 39 MB/s | 3.1 MB 00:00 2021-11-20T05:55:18.724 INFO:teuthology.orchestra.run.smithi178.stdout:(20/22): slirp4netns-1.1.8-1.module_el8.6.0+926 3.5 MB/s | 51 kB 00:00 2021-11-20T05:55:18.763 INFO:teuthology.orchestra.run.smithi178.stdout:(21/22): skopeo-1.5.0-2.module_el8.6.0+954+963c 18 MB/s | 6.7 MB 00:00 2021-11-20T05:55:18.892 INFO:teuthology.orchestra.run.smithi139.stdout:(22/22): containernetworking-plugins-1.0.1-1.mo 33 MB/s | 19 MB 00:00 2021-11-20T05:55:18.896 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:55:18.896 INFO:teuthology.orchestra.run.smithi139.stdout:Total 53 MB/s | 41 MB 00:00 2021-11-20T05:55:18.930 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:55:18.957 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:55:18.957 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:55:19.045 INFO:teuthology.orchestra.run.smithi178.stdout:(22/22): containernetworking-plugins-1.0.1-1.mo 43 MB/s | 19 MB 00:00 2021-11-20T05:55:19.048 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:55:19.049 INFO:teuthology.orchestra.run.smithi178.stdout:Total 53 MB/s | 41 MB 00:00 2021-11-20T05:55:19.082 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:55:19.109 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:55:19.109 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:55:19.297 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:55:19.300 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:55:19.447 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:55:19.451 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:55:20.365 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:55:20.529 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 1/29 2021-11-20T05:55:20.557 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:55:20.688 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 1/29 2021-11-20T05:55:20.717 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 2/29 2021-11-20T05:55:20.870 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 2/29 2021-11-20T05:55:21.233 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 3/29 2021-11-20T05:55:21.410 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 3/29 2021-11-20T05:55:21.422 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 4/29 2021-11-20T05:55:21.574 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 4/29 2021-11-20T05:55:21.608 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 5/29 2021-11-20T05:55:21.752 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 5/29 2021-11-20T05:55:21.792 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 6/29 2021-11-20T05:55:21.969 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 6/29 2021-11-20T05:55:21.998 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-idna-2.5-5.el8.noarch 7/29 2021-11-20T05:55:22.152 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-idna-2.5-5.el8.noarch 7/29 2021-11-20T05:55:22.211 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 8/29 2021-11-20T05:55:22.364 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 8/29 2021-11-20T05:55:22.379 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 9/29 2021-11-20T05:55:22.505 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 9/29 2021-11-20T05:55:22.536 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T05:55:22.663 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T05:55:22.706 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 11/29 2021-11-20T05:55:22.854 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 11/29 2021-11-20T05:55:22.899 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 12/29 2021-11-20T05:55:23.050 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 12/29 2021-11-20T05:55:23.088 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 13/29 2021-11-20T05:55:23.240 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 13/29 2021-11-20T05:55:23.921 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 14/29 2021-11-20T05:55:24.029 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 14/29 2021-11-20T05:55:24.136 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 15/29 2021-11-20T05:55:24.224 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 15/29 2021-11-20T05:55:24.272 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T05:55:24.349 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T05:55:25.879 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T05:55:25.952 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-20T05:55:26.095 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : containernetworking-plugins-1.0.1-1.module_el8.6.0 17/29 2021-11-20T05:55:26.113 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : containernetworking-plugins-1.0.1-1.module_el8.6.0 17/29 2021-11-20T05:55:26.270 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 18/29 2021-11-20T05:55:26.293 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 18/29 2021-11-20T05:55:26.646 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 19/29 2021-11-20T05:55:26.669 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 19/29 2021-11-20T05:55:27.396 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 20/29 2021-11-20T05:55:27.402 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 20/29 2021-11-20T05:55:27.552 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 21/29 2021-11-20T05:55:27.559 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 21/29 2021-11-20T05:55:27.737 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 22/29 2021-11-20T05:55:27.743 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 22/29 2021-11-20T05:55:27.883 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 23/29 2021-11-20T05:55:27.889 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 23/29 2021-11-20T05:55:28.007 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 24/29 2021-11-20T05:55:28.056 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 24/29 2021-11-20T05:55:28.174 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 25/29 2021-11-20T05:55:28.212 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 25/29 2021-11-20T05:55:28.299 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 26/29 2021-11-20T05:55:28.355 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 26/29 2021-11-20T05:55:28.442 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 27/29 2021-11-20T05:55:28.498 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 27/29 2021-11-20T05:55:28.624 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : containernetworking-plugins-0.9.1-1.module_el8.5.0 28/29 2021-11-20T05:55:28.705 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : containernetworking-plugins-0.9.1-1.module_el8.5.0 28/29 2021-11-20T05:55:28.780 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T05:55:28.853 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T05:55:29.226 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T05:55:29.227 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 1/29 2021-11-20T05:55:29.227 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 2/29 2021-11-20T05:55:29.227 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 3/29 2021-11-20T05:55:29.227 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/29 2021-11-20T05:55:29.228 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 5/29 2021-11-20T05:55:29.228 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 6/29 2021-11-20T05:55:29.228 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 7/29 2021-11-20T05:55:29.228 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 8/29 2021-11-20T05:55:29.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 9/29 2021-11-20T05:55:29.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T05:55:29.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 11/29 2021-11-20T05:55:29.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-idna-2.5-5.el8.noarch 12/29 2021-11-20T05:55:29.229 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 13/29 2021-11-20T05:55:29.230 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 14/29 2021-11-20T05:55:29.230 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 15/29 2021-11-20T05:55:29.231 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 16/29 2021-11-20T05:55:29.232 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 17/29 2021-11-20T05:55:29.232 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : containernetworking-plugins-1.0.1-1.module_el8.6.0 18/29 2021-11-20T05:55:29.232 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : containernetworking-plugins-0.9.1-1.module_el8.5.0 19/29 2021-11-20T05:55:29.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 20/29 2021-11-20T05:55:29.233 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 21/29 2021-11-20T05:55:29.234 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 22/29 2021-11-20T05:55:29.234 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 23/29 2021-11-20T05:55:29.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 24/29 2021-11-20T05:55:29.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 25/29 2021-11-20T05:55:29.235 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 26/29 2021-11-20T05:55:29.236 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 27/29 2021-11-20T05:55:29.236 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 28/29 2021-11-20T05:55:29.246 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-20T05:55:29.247 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 1/29 2021-11-20T05:55:29.247 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 2/29 2021-11-20T05:55:29.247 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 3/29 2021-11-20T05:55:29.248 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/29 2021-11-20T05:55:29.248 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 5/29 2021-11-20T05:55:29.249 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 6/29 2021-11-20T05:55:29.249 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 7/29 2021-11-20T05:55:29.250 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 8/29 2021-11-20T05:55:29.250 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 9/29 2021-11-20T05:55:29.251 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-20T05:55:29.251 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 11/29 2021-11-20T05:55:29.252 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-idna-2.5-5.el8.noarch 12/29 2021-11-20T05:55:29.252 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 13/29 2021-11-20T05:55:29.253 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 14/29 2021-11-20T05:55:29.253 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 15/29 2021-11-20T05:55:29.253 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 16/29 2021-11-20T05:55:29.254 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 17/29 2021-11-20T05:55:29.254 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : containernetworking-plugins-1.0.1-1.module_el8.6.0 18/29 2021-11-20T05:55:29.255 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : containernetworking-plugins-0.9.1-1.module_el8.5.0 19/29 2021-11-20T05:55:29.255 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 20/29 2021-11-20T05:55:29.256 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 21/29 2021-11-20T05:55:29.256 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 22/29 2021-11-20T05:55:29.256 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 23/29 2021-11-20T05:55:29.257 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 24/29 2021-11-20T05:55:29.257 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 25/29 2021-11-20T05:55:29.258 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 26/29 2021-11-20T05:55:29.258 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 27/29 2021-11-20T05:55:29.259 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 28/29 2021-11-20T05:55:29.909 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 29/29 2021-11-20T05:55:29.910 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:55:29.910 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2021-11-20T05:55:29.910 INFO:teuthology.orchestra.run.smithi178.stdout: conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T05:55:29.910 INFO:teuthology.orchestra.run.smithi178.stdout: containernetworking-plugins-1.0.1-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T05:55:29.911 INFO:teuthology.orchestra.run.smithi178.stdout: criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:55:29.911 INFO:teuthology.orchestra.run.smithi178.stdout: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:55:29.911 INFO:teuthology.orchestra.run.smithi178.stdout: libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:55:29.911 INFO:teuthology.orchestra.run.smithi178.stdout: runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:55:29.912 INFO:teuthology.orchestra.run.smithi178.stdout: slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:55:29.912 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:55:29.912 INFO:teuthology.orchestra.run.smithi178.stdout: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T05:55:29.912 INFO:teuthology.orchestra.run.smithi178.stdout: cockpit-podman-35-1.module_el8.6.0+944+d413f95e.noarch 2021-11-20T05:55:29.912 INFO:teuthology.orchestra.run.smithi178.stdout: crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 2021-11-20T05:55:29.913 INFO:teuthology.orchestra.run.smithi178.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-11-20T05:55:29.913 INFO:teuthology.orchestra.run.smithi178.stdout: python3-idna-2.5-5.el8.noarch 2021-11-20T05:55:29.913 INFO:teuthology.orchestra.run.smithi178.stdout: python3-podman-3.2.1-4.module_el8.6.0+954+963caf36.noarch 2021-11-20T05:55:29.913 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-11-20T05:55:29.914 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2021-11-20T05:55:29.914 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyxdg-0.25-16.el8.noarch 2021-11-20T05:55:29.914 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-11-20T05:55:29.915 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-11-20T05:55:29.915 INFO:teuthology.orchestra.run.smithi178.stdout: skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T05:55:29.915 INFO:teuthology.orchestra.run.smithi178.stdout: toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86_64 2021-11-20T05:55:29.915 INFO:teuthology.orchestra.run.smithi178.stdout: udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 2021-11-20T05:55:29.915 INFO:teuthology.orchestra.run.smithi178.stdout: yajl-2.1.0-10.el8.x86_64 2021-11-20T05:55:29.916 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:55:29.916 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:55:29.925 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 29/29 2021-11-20T05:55:29.926 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:55:29.926 INFO:teuthology.orchestra.run.smithi139.stdout:Upgraded: 2021-11-20T05:55:29.926 INFO:teuthology.orchestra.run.smithi139.stdout: conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T05:55:29.927 INFO:teuthology.orchestra.run.smithi139.stdout: containernetworking-plugins-1.0.1-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-20T05:55:29.927 INFO:teuthology.orchestra.run.smithi139.stdout: criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:55:29.927 INFO:teuthology.orchestra.run.smithi139.stdout: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:55:29.927 INFO:teuthology.orchestra.run.smithi139.stdout: libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:55:29.927 INFO:teuthology.orchestra.run.smithi139.stdout: runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:55:29.928 INFO:teuthology.orchestra.run.smithi139.stdout: slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-20T05:55:29.928 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:55:29.928 INFO:teuthology.orchestra.run.smithi139.stdout: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T05:55:29.929 INFO:teuthology.orchestra.run.smithi139.stdout: cockpit-podman-35-1.module_el8.6.0+944+d413f95e.noarch 2021-11-20T05:55:29.929 INFO:teuthology.orchestra.run.smithi139.stdout: crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 2021-11-20T05:55:29.930 INFO:teuthology.orchestra.run.smithi139.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-11-20T05:55:29.930 INFO:teuthology.orchestra.run.smithi139.stdout: python3-idna-2.5-5.el8.noarch 2021-11-20T05:55:29.930 INFO:teuthology.orchestra.run.smithi139.stdout: python3-podman-3.2.1-4.module_el8.6.0+954+963caf36.noarch 2021-11-20T05:55:29.930 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-11-20T05:55:29.931 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2021-11-20T05:55:29.931 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyxdg-0.25-16.el8.noarch 2021-11-20T05:55:29.931 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-11-20T05:55:29.931 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-11-20T05:55:29.931 INFO:teuthology.orchestra.run.smithi139.stdout: skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_64 2021-11-20T05:55:29.932 INFO:teuthology.orchestra.run.smithi139.stdout: toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86_64 2021-11-20T05:55:29.932 INFO:teuthology.orchestra.run.smithi139.stdout: udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 2021-11-20T05:55:29.932 INFO:teuthology.orchestra.run.smithi139.stdout: yajl-2.1.0-10.el8.x86_64 2021-11-20T05:55:29.932 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:55:29.933 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:55:30.165 DEBUG:teuthology.parallel:result is None 2021-11-20T05:55:30.193 DEBUG:teuthology.parallel:result is None 2021-11-20T05:55:30.194 INFO:teuthology.run_tasks:Running task install... 2021-11-20T05:55:30.205 DEBUG:teuthology.task.install:project ceph 2021-11-20T05:55:30.206 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa'}} 2021-11-20T05:55:30.206 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa'} 2021-11-20T05:55:30.207 INFO:teuthology.task.install:Using flavor: default 2021-11-20T05:55:30.215 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-11-20T05:55:30.215 INFO:teuthology.task.install:extra packages: [] 2021-11-20T05:55:30.216 DEBUG:teuthology.orchestra.run.smithi139:> sudo lsb_release -is 2021-11-20T05:55:30.250 INFO:teuthology.orchestra.run.smithi139.stdout:CentOSStream 2021-11-20T05:55:30.251 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T05:55:30.252 DEBUG:teuthology.orchestra.run.smithi178:> sudo lsb_release -is 2021-11-20T05:55:30.253 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa', 'tag': 'v16.2.4', 'wait_for_package': False} 2021-11-20T05:55:30.254 DEBUG:teuthology.orchestra.run.smithi139:> sudo lsb_release -is 2021-11-20T05:55:30.288 INFO:teuthology.orchestra.run.smithi178.stdout:CentOSStream 2021-11-20T05:55:30.290 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T05:55:30.290 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa', 'tag': 'v16.2.4', 'wait_for_package': False} 2021-11-20T05:55:30.291 DEBUG:teuthology.orchestra.run.smithi178:> sudo lsb_release -is 2021-11-20T05:55:30.324 INFO:teuthology.orchestra.run.smithi139.stdout:CentOSStream 2021-11-20T05:55:30.325 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T05:55:30.326 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T05:55:30.326 INFO:teuthology.packaging:ref: None 2021-11-20T05:55:30.326 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T05:55:30.327 INFO:teuthology.packaging:branch: None 2021-11-20T05:55:30.327 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T05:55:30.645 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:55:30.646 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:55:30.697 INFO:teuthology.orchestra.run.smithi178.stdout:CentOSStream 2021-11-20T05:55:30.697 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T05:55:30.698 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T05:55:30.698 INFO:teuthology.packaging:ref: None 2021-11-20T05:55:30.699 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T05:55:30.699 INFO:teuthology.packaging:branch: None 2021-11-20T05:55:30.699 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T05:55:30.915 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:55:30.916 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:55:31.092 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2021-11-20T05:55:31.092 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2021-11-20T05:55:31.128 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2021-11-20T05:55:31.129 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2021-11-20T05:55:31.140 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-11-20T05:55:31.141 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-11-20T05:55:31.141 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-11-20T05:55:31.157 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-11-20T05:55:31.157 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2021-11-20T05:55:31.158 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-11-20T05:55:31.169 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-11-20T05:55:31.170 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T05:55:31.170 INFO:teuthology.packaging:ref: None 2021-11-20T05:55:31.170 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T05:55:31.171 INFO:teuthology.packaging:branch: None 2021-11-20T05:55:31.171 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T05:55:31.171 DEBUG:teuthology.orchestra.run.smithi139:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-11-20T05:55:31.186 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-11-20T05:55:31.187 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T05:55:31.187 INFO:teuthology.packaging:ref: None 2021-11-20T05:55:31.187 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T05:55:31.187 INFO:teuthology.packaging:branch: None 2021-11-20T05:55:31.188 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T05:55:31.188 DEBUG:teuthology.orchestra.run.smithi178:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-11-20T05:55:31.237 DEBUG:teuthology.orchestra.run.smithi139:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-11-20T05:55:31.254 DEBUG:teuthology.orchestra.run.smithi178:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-11-20T05:55:31.314 DEBUG:teuthology.orchestra.run.smithi139:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-11-20T05:55:31.331 DEBUG:teuthology.orchestra.run.smithi178:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-11-20T05:55:31.379 INFO:teuthology.orchestra.run.smithi139.stdout:check_obsoletes = 1 2021-11-20T05:55:31.381 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum clean all 2021-11-20T05:55:31.396 INFO:teuthology.orchestra.run.smithi178.stdout:check_obsoletes = 1 2021-11-20T05:55:31.397 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum clean all 2021-11-20T05:55:31.616 INFO:teuthology.orchestra.run.smithi139.stdout:75 files removed 2021-11-20T05:55:31.633 INFO:teuthology.orchestra.run.smithi178.stdout:75 files removed 2021-11-20T05:55:31.636 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-radosgw 2021-11-20T05:55:31.653 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw 2021-11-20T05:55:32.001 INFO:teuthology.orchestra.run.smithi139.stdout:ceph packages for x86_64 2.7 MB/s | 74 kB 00:00 2021-11-20T05:55:32.015 INFO:teuthology.orchestra.run.smithi178.stdout:ceph packages for x86_64 3.0 MB/s | 74 kB 00:00 2021-11-20T05:55:32.061 INFO:teuthology.orchestra.run.smithi139.stdout:ceph noarch packages 615 kB/s | 15 kB 00:00 2021-11-20T05:55:32.074 INFO:teuthology.orchestra.run.smithi178.stdout:ceph noarch packages 653 kB/s | 15 kB 00:00 2021-11-20T05:55:32.113 INFO:teuthology.orchestra.run.smithi139.stdout:ceph source packages 73 kB/s | 1.7 kB 00:00 2021-11-20T05:55:32.126 INFO:teuthology.orchestra.run.smithi178.stdout:ceph source packages 73 kB/s | 1.7 kB 00:00 2021-11-20T05:55:32.543 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - AppStream 42 MB/s | 17 MB 00:00 2021-11-20T05:55:32.637 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-8 - AppStream 35 MB/s | 17 MB 00:00 2021-11-20T05:55:35.782 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - Base 43 MB/s | 15 MB 00:00 2021-11-20T05:55:35.788 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-8 - Base 51 MB/s | 15 MB 00:00 2021-11-20T05:55:37.989 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - Extras 235 kB/s | 16 kB 00:00 2021-11-20T05:55:38.008 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-8 - Extras 225 kB/s | 16 kB 00:00 2021-11-20T05:55:38.198 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS-8 - PowerTools 33 MB/s | 3.6 MB 00:00 2021-11-20T05:55:38.229 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-8 - PowerTools 30 MB/s | 3.6 MB 00:00 2021-11-20T05:55:41.296 INFO:teuthology.orchestra.run.smithi178.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2021-11-20T05:55:41.304 INFO:teuthology.orchestra.run.smithi139.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2021-11-20T05:55:42.660 INFO:teuthology.orchestra.run.smithi178.stdout:Apache Arrow 146 kB/s | 184 kB 00:01 2021-11-20T05:55:43.001 INFO:teuthology.orchestra.run.smithi139.stdout:Apache Arrow 115 kB/s | 184 kB 00:01 2021-11-20T05:55:43.563 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 14 MB/s | 11 MB 00:00 2021-11-20T05:55:44.048 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 12 MB/s | 11 MB 00:00 2021-11-20T05:55:45.921 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2021-11-20T05:55:46.449 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2021-11-20T05:55:48.668 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:55:48.670 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:55:48.670 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2021-11-20T05:55:48.671 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:55:48.671 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:55:48.671 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2021-11-20T05:55:48.671 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2021-11-20T05:55:48.672 INFO:teuthology.orchestra.run.smithi178.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T05:55:48.672 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2021-11-20T05:55:48.672 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2021-11-20T05:55:48.672 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2021-11-20T05:55:48.672 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2021-11-20T05:55:48.673 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2021-11-20T05:55:48.673 INFO:teuthology.orchestra.run.smithi178.stdout: daxctl-libs x86_64 71.1-2.el8 CentOS-Base 41 k 2021-11-20T05:55:48.673 INFO:teuthology.orchestra.run.smithi178.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-11-20T05:55:48.673 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-11-20T05:55:48.674 INFO:teuthology.orchestra.run.smithi178.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-11-20T05:55:48.674 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2021-11-20T05:55:48.674 INFO:teuthology.orchestra.run.smithi178.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-11-20T05:55:48.674 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj x86_64 1.6.1-1.el8 CentOS-AppStream 145 k 2021-11-20T05:55:48.674 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-11-20T05:55:48.675 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2021-11-20T05:55:48.675 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2021-11-20T05:55:48.675 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2021-11-20T05:55:48.675 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-11-20T05:55:48.675 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-11-20T05:55:48.676 INFO:teuthology.orchestra.run.smithi178.stdout: ndctl-libs x86_64 71.1-2.el8 CentOS-Base 78 k 2021-11-20T05:55:48.676 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2021-11-20T05:55:48.676 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2021-11-20T05:55:48.676 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2021-11-20T05:55:48.677 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-11-20T05:55:48.677 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2021-11-20T05:55:48.677 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2021-11-20T05:55:48.677 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2021-11-20T05:55:48.677 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:55:48.678 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:55:48.678 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:55:48.678 INFO:teuthology.orchestra.run.smithi178.stdout:Install 25 Packages 2021-11-20T05:55:48.678 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 2 Packages 2021-11-20T05:55:48.679 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:55:48.679 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 59 M 2021-11-20T05:55:48.680 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:55:48.822 INFO:teuthology.orchestra.run.smithi178.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 41 MB/s | 6.0 MB 00:00 2021-11-20T05:55:48.829 INFO:teuthology.orchestra.run.smithi178.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 3.4 MB/s | 25 kB 00:00 2021-11-20T05:55:48.851 INFO:teuthology.orchestra.run.smithi178.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 38 MB/s | 801 kB 00:00 2021-11-20T05:55:48.867 INFO:teuthology.orchestra.run.smithi178.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 29 MB/s | 486 kB 00:00 2021-11-20T05:55:48.977 INFO:teuthology.orchestra.run.smithi178.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 37 MB/s | 4.1 MB 00:00 2021-11-20T05:55:48.988 INFO:teuthology.orchestra.run.smithi178.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 4.2 MB/s | 44 kB 00:00 2021-11-20T05:55:48.998 INFO:teuthology.orchestra.run.smithi178.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 7.8 MB/s | 80 kB 00:00 2021-11-20T05:55:49.011 INFO:teuthology.orchestra.run.smithi178.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 17 MB/s | 213 kB 00:00 2021-11-20T05:55:49.026 INFO:teuthology.orchestra.run.smithi178.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 25 MB/s | 387 kB 00:00 2021-11-20T05:55:49.042 INFO:teuthology.orchestra.run.smithi178.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 22 MB/s | 368 kB 00:00 2021-11-20T05:55:49.056 INFO:teuthology.orchestra.run.smithi178.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 8.7 MB/s | 114 kB 00:00 2021-11-20T05:55:49.184 INFO:teuthology.orchestra.run.smithi178.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 25 MB/s | 13 MB 00:00 2021-11-20T05:55:49.192 INFO:teuthology.orchestra.run.smithi178.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 1.0 MB/s | 145 kB 00:00 2021-11-20T05:55:49.195 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:55:49.197 INFO:teuthology.orchestra.run.smithi178.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 54 MB/s | 259 kB 00:00 2021-11-20T05:55:49.198 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:55:49.198 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-11-20T05:55:49.199 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:55:49.199 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:55:49.199 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2021-11-20T05:55:49.199 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrading: 2021-11-20T05:55:49.200 INFO:teuthology.orchestra.run.smithi139.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T05:55:49.200 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2021-11-20T05:55:49.200 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-11-20T05:55:49.200 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2021-11-20T05:55:49.201 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2021-11-20T05:55:49.201 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2021-11-20T05:55:49.201 INFO:teuthology.orchestra.run.smithi139.stdout: daxctl-libs x86_64 71.1-2.el8 CentOS-Base 41 k 2021-11-20T05:55:49.201 INFO:teuthology.orchestra.run.smithi139.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-11-20T05:55:49.201 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-11-20T05:55:49.202 INFO:teuthology.orchestra.run.smithi139.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-11-20T05:55:49.202 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2021-11-20T05:55:49.202 INFO:teuthology.orchestra.run.smithi139.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-11-20T05:55:49.202 INFO:teuthology.orchestra.run.smithi139.stdout: libpmemobj x86_64 1.6.1-1.el8 CentOS-AppStream 145 k 2021-11-20T05:55:49.203 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-11-20T05:55:49.203 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2021-11-20T05:55:49.203 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2021-11-20T05:55:49.203 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2021-11-20T05:55:49.204 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-11-20T05:55:49.204 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-11-20T05:55:49.204 INFO:teuthology.orchestra.run.smithi139.stdout: ndctl-libs x86_64 71.1-2.el8 CentOS-Base 78 k 2021-11-20T05:55:49.204 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2021-11-20T05:55:49.205 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2021-11-20T05:55:49.205 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2021-11-20T05:55:49.205 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-11-20T05:55:49.205 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2021-11-20T05:55:49.205 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2021-11-20T05:55:49.206 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2021-11-20T05:55:49.206 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:55:49.206 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:55:49.206 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:55:49.207 INFO:teuthology.orchestra.run.smithi139.stdout:Install 25 Packages 2021-11-20T05:55:49.207 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrade 2 Packages 2021-11-20T05:55:49.207 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:55:49.208 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 59 M 2021-11-20T05:55:49.209 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:55:49.209 INFO:teuthology.orchestra.run.smithi178.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 33 MB/s | 44 kB 00:00 2021-11-20T05:55:49.209 INFO:teuthology.orchestra.run.smithi178.stdout:(16/27): daxctl-libs-71.1-2.el8.x86_64.rpm 30 MB/s | 41 kB 00:00 2021-11-20T05:55:49.209 INFO:teuthology.orchestra.run.smithi178.stdout:(17/27): librabbitmq-0.9.0-3.el8.x86_64.rpm 34 MB/s | 47 kB 00:00 2021-11-20T05:55:49.210 INFO:teuthology.orchestra.run.smithi178.stdout:(18/27): ndctl-libs-71.1-2.el8.x86_64.rpm 46 MB/s | 78 kB 00:00 2021-11-20T05:55:49.217 INFO:teuthology.orchestra.run.smithi178.stdout:(19/27): librdkafka-0.11.4-3.el8.x86_64.rpm 10 MB/s | 354 kB 00:00 2021-11-20T05:55:49.376 INFO:teuthology.orchestra.run.smithi178.stdout:(20/27): ceph-common-16.2.4-0.el8.x86_64.rpm 34 MB/s | 24 MB 00:00 2021-11-20T05:55:49.390 INFO:teuthology.orchestra.run.smithi139.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 32 MB/s | 6.0 MB 00:00 2021-11-20T05:55:49.391 INFO:teuthology.orchestra.run.smithi139.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 15 MB/s | 25 kB 00:00 2021-11-20T05:55:49.413 INFO:teuthology.orchestra.run.smithi139.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 37 MB/s | 801 kB 00:00 2021-11-20T05:55:49.427 INFO:teuthology.orchestra.run.smithi139.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 35 MB/s | 486 kB 00:00 2021-11-20T05:55:49.443 INFO:teuthology.orchestra.run.smithi178.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 487 kB/s | 116 kB 00:00 2021-11-20T05:55:49.472 INFO:teuthology.orchestra.run.smithi178.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 2.0 MB/s | 59 kB 00:00 2021-11-20T05:55:49.502 INFO:teuthology.orchestra.run.smithi178.stdout:(23/27): libunwind-1.3.1-3.el8.x86_64.rpm 2.5 MB/s | 75 kB 00:00 2021-11-20T05:55:49.519 INFO:teuthology.orchestra.run.smithi178.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2021-11-20T05:55:49.554 INFO:teuthology.orchestra.run.smithi178.stdout:(25/27): leveldb-1.22-1.el8.x86_64.rpm 1.0 MB/s | 181 kB 00:00 2021-11-20T05:55:49.560 INFO:teuthology.orchestra.run.smithi139.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 31 MB/s | 4.1 MB 00:00 2021-11-20T05:55:49.562 INFO:teuthology.orchestra.run.smithi139.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 21 MB/s | 44 kB 00:00 2021-11-20T05:55:49.565 INFO:teuthology.orchestra.run.smithi139.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 36 MB/s | 80 kB 00:00 2021-11-20T05:55:49.584 INFO:teuthology.orchestra.run.smithi139.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 11 MB/s | 213 kB 00:00 2021-11-20T05:55:49.607 INFO:teuthology.orchestra.run.smithi178.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 46 MB/s | 4.0 MB 00:00 2021-11-20T05:55:49.607 INFO:teuthology.orchestra.run.smithi139.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 16 MB/s | 387 kB 00:00 2021-11-20T05:55:49.619 INFO:teuthology.orchestra.run.smithi139.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 31 MB/s | 368 kB 00:00 2021-11-20T05:55:49.622 INFO:teuthology.orchestra.run.smithi178.stdout:(27/27): librados2-16.2.4-0.el8.x86_64.rpm 31 MB/s | 3.7 MB 00:00 2021-11-20T05:55:49.623 INFO:teuthology.orchestra.run.smithi139.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 33 MB/s | 114 kB 00:00 2021-11-20T05:55:49.626 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:55:49.628 INFO:teuthology.orchestra.run.smithi178.stdout:Total 61 MB/s | 59 MB 00:00 2021-11-20T05:55:49.638 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:55:49.657 INFO:teuthology.orchestra.run.smithi139.stdout:(12/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 4.2 MB/s | 145 kB 00:00 2021-11-20T05:55:49.662 INFO:teuthology.orchestra.run.smithi139.stdout:(13/27): librdkafka-0.11.4-3.el8.x86_64.rpm 72 MB/s | 354 kB 00:00 2021-11-20T05:55:49.666 INFO:teuthology.orchestra.run.smithi139.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 68 MB/s | 259 kB 00:00 2021-11-20T05:55:49.667 INFO:teuthology.orchestra.run.smithi139.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 32 MB/s | 44 kB 00:00 2021-11-20T05:55:49.669 INFO:teuthology.orchestra.run.smithi139.stdout:(16/27): daxctl-libs-71.1-2.el8.x86_64.rpm 31 MB/s | 41 kB 00:00 2021-11-20T05:55:49.671 INFO:teuthology.orchestra.run.smithi139.stdout:(17/27): librabbitmq-0.9.0-3.el8.x86_64.rpm 33 MB/s | 47 kB 00:00 2021-11-20T05:55:49.673 INFO:teuthology.orchestra.run.smithi139.stdout:(18/27): ndctl-libs-71.1-2.el8.x86_64.rpm 45 MB/s | 78 kB 00:00 2021-11-20T05:55:49.685 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:55:49.685 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:55:49.777 INFO:teuthology.orchestra.run.smithi139.stdout:(19/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 22 MB/s | 13 MB 00:00 2021-11-20T05:55:49.987 INFO:teuthology.orchestra.run.smithi139.stdout:(20/27): ceph-common-16.2.4-0.el8.x86_64.rpm 31 MB/s | 24 MB 00:00 2021-11-20T05:55:50.005 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:55:50.009 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:55:50.056 INFO:teuthology.orchestra.run.smithi139.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 305 kB/s | 116 kB 00:00 2021-11-20T05:55:50.084 INFO:teuthology.orchestra.run.smithi139.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2021-11-20T05:55:50.085 INFO:teuthology.orchestra.run.smithi139.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 1.9 MB/s | 59 kB 00:00 2021-11-20T05:55:50.113 INFO:teuthology.orchestra.run.smithi139.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 2.4 MB/s | 75 kB 00:00 2021-11-20T05:55:50.131 INFO:teuthology.orchestra.run.smithi139.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 81 MB/s | 3.7 MB 00:00 2021-11-20T05:55:50.174 INFO:teuthology.orchestra.run.smithi139.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 68 MB/s | 4.0 MB 00:00 2021-11-20T05:55:50.175 INFO:teuthology.orchestra.run.smithi139.stdout:(27/27): leveldb-1.22-1.el8.x86_64.rpm 962 kB/s | 181 kB 00:00 2021-11-20T05:55:50.179 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:55:50.179 INFO:teuthology.orchestra.run.smithi139.stdout:Total 60 MB/s | 59 MB 00:00 2021-11-20T05:55:50.190 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:55:50.238 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:55:50.238 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:55:50.564 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:55:50.567 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:55:51.154 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:55:51.308 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2021-11-20T05:55:51.462 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2021-11-20T05:55:51.628 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T05:55:51.669 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:55:51.848 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2021-11-20T05:55:52.035 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2021-11-20T05:55:52.039 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T05:55:52.203 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T05:55:52.210 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T05:55:52.309 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T05:55:52.552 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T05:55:52.623 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-20T05:55:52.685 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T05:55:52.753 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T05:55:52.784 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T05:55:52.885 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T05:55:52.942 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/29 2021-11-20T05:55:53.085 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T05:55:53.091 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T05:55:53.225 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T05:55:53.289 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T05:55:53.457 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/29 2021-11-20T05:55:53.512 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T05:55:53.608 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T05:55:53.651 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T05:55:53.727 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T05:55:53.889 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2021-11-20T05:55:54.001 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T05:55:54.028 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-20T05:55:54.051 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T05:55:54.167 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T05:55:54.173 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T05:55:54.245 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T05:55:54.327 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T05:55:54.407 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2021-11-20T05:55:54.460 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T05:55:54.517 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T05:55:54.525 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T05:55:54.612 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-20T05:55:54.693 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T05:55:54.747 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T05:55:54.894 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T05:55:55.033 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T05:55:55.093 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-20T05:55:55.106 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T05:55:55.250 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T05:55:55.267 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T05:55:55.366 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T05:55:55.564 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2021-11-20T05:55:55.682 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-20T05:55:55.745 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2021-11-20T05:55:55.851 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T05:55:55.858 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T05:55:55.925 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T05:55:55.951 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-20T05:55:56.109 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2021-11-20T05:55:56.120 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2021-11-20T05:55:56.276 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2021-11-20T05:55:56.301 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2021-11-20T05:55:56.467 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T05:55:56.529 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2021-11-20T05:55:56.535 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-20T05:55:56.716 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2021-11-20T05:55:56.735 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2021-11-20T05:55:57.544 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2021-11-20T05:55:57.759 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2021-11-20T05:55:57.942 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2021-11-20T05:56:01.746 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T05:56:01.942 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T05:56:02.265 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T05:56:02.482 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T05:56:02.645 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T05:56:02.917 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T05:56:03.010 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T05:56:03.176 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-20T05:56:03.366 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T05:56:03.431 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T05:56:03.520 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-20T05:56:03.665 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T05:56:21.215 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T05:56:21.215 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /sys 2021-11-20T05:56:21.216 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /proc 2021-11-20T05:56:21.216 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /dev 2021-11-20T05:56:21.216 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /run 2021-11-20T05:56:21.216 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /mnt 2021-11-20T05:56:21.217 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /var/tmp 2021-11-20T05:56:21.217 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /home 2021-11-20T05:56:21.217 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /tmp 2021-11-20T05:56:21.217 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /dev 2021-11-20T05:56:21.218 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:21.447 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T05:56:21.447 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /sys 2021-11-20T05:56:21.448 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /proc 2021-11-20T05:56:21.448 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /dev 2021-11-20T05:56:21.448 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /run 2021-11-20T05:56:21.448 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /mnt 2021-11-20T05:56:21.448 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /var/tmp 2021-11-20T05:56:21.449 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /home 2021-11-20T05:56:21.449 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /tmp 2021-11-20T05:56:21.449 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /dev 2021-11-20T05:56:21.449 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:22.532 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T05:56:22.605 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T05:56:22.605 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T05:56:22.735 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T05:56:22.798 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T05:56:22.798 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T05:56:22.807 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-20T05:56:22.808 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T05:56:23.005 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-20T05:56:23.006 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T05:56:23.542 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T05:56:23.542 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2021-11-20T05:56:23.542 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2021-11-20T05:56:23.542 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2021-11-20T05:56:23.543 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T05:56:23.543 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T05:56:23.543 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T05:56:23.543 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2021-11-20T05:56:23.544 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2021-11-20T05:56:23.544 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T05:56:23.544 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2021-11-20T05:56:23.544 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2021-11-20T05:56:23.545 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T05:56:23.545 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T05:56:23.545 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2021-11-20T05:56:23.545 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2021-11-20T05:56:23.545 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2021-11-20T05:56:23.546 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2021-11-20T05:56:23.546 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : daxctl-libs-71.1-2.el8.x86_64 18/29 2021-11-20T05:56:23.546 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 19/29 2021-11-20T05:56:23.546 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ndctl-libs-71.1-2.el8.x86_64 20/29 2021-11-20T05:56:23.547 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2021-11-20T05:56:23.547 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2021-11-20T05:56:23.547 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2021-11-20T05:56:23.547 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2021-11-20T05:56:23.548 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2021-11-20T05:56:23.548 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T05:56:23.548 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2021-11-20T05:56:23.548 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2021-11-20T05:56:23.724 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T05:56:23.724 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2021-11-20T05:56:23.724 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2021-11-20T05:56:23.725 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2021-11-20T05:56:23.725 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2021-11-20T05:56:23.725 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2021-11-20T05:56:23.725 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2021-11-20T05:56:23.726 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2021-11-20T05:56:23.726 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2021-11-20T05:56:23.726 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2021-11-20T05:56:23.726 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2021-11-20T05:56:23.727 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2021-11-20T05:56:23.727 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2021-11-20T05:56:23.727 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2021-11-20T05:56:23.727 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2021-11-20T05:56:23.728 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2021-11-20T05:56:23.728 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2021-11-20T05:56:23.728 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2021-11-20T05:56:23.728 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : daxctl-libs-71.1-2.el8.x86_64 18/29 2021-11-20T05:56:23.729 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 19/29 2021-11-20T05:56:23.729 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ndctl-libs-71.1-2.el8.x86_64 20/29 2021-11-20T05:56:23.729 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2021-11-20T05:56:23.729 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2021-11-20T05:56:23.730 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2021-11-20T05:56:23.730 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2021-11-20T05:56:23.730 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2021-11-20T05:56:23.730 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2021-11-20T05:56:23.731 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2021-11-20T05:56:23.731 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2021-11-20T05:56:24.511 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T05:56:24.512 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:24.512 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2021-11-20T05:56:24.512 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.512 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:56:24.513 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.513 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.513 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.513 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.514 INFO:teuthology.orchestra.run.smithi178.stdout: daxctl-libs-71.1-2.el8.x86_64 2021-11-20T05:56:24.514 INFO:teuthology.orchestra.run.smithi178.stdout: fmt-6.2.1-1.el8.x86_64 2021-11-20T05:56:24.514 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-11-20T05:56:24.521 INFO:teuthology.orchestra.run.smithi178.stdout: leveldb-1.22-1.el8.x86_64 2021-11-20T05:56:24.522 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.522 INFO:teuthology.orchestra.run.smithi178.stdout: liboath-2.6.2-3.el8.x86_64 2021-11-20T05:56:24.522 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj-1.6.1-1.el8.x86_64 2021-11-20T05:56:24.522 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-11-20T05:56:24.523 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.523 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka-0.11.4-3.el8.x86_64 2021-11-20T05:56:24.523 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.523 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind-1.3.1-3.el8.x86_64 2021-11-20T05:56:24.524 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-11-20T05:56:24.524 INFO:teuthology.orchestra.run.smithi178.stdout: ndctl-libs-71.1-2.el8.x86_64 2021-11-20T05:56:24.524 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.524 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.524 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.525 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-11-20T05:56:24.525 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.525 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.525 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.526 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:24.526 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:56:24.616 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2021-11-20T05:56:24.616 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:24.617 INFO:teuthology.orchestra.run.smithi139.stdout:Upgraded: 2021-11-20T05:56:24.617 INFO:teuthology.orchestra.run.smithi139.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.617 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:56:24.618 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.618 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.618 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.618 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.618 INFO:teuthology.orchestra.run.smithi139.stdout: daxctl-libs-71.1-2.el8.x86_64 2021-11-20T05:56:24.619 INFO:teuthology.orchestra.run.smithi139.stdout: fmt-6.2.1-1.el8.x86_64 2021-11-20T05:56:24.619 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-11-20T05:56:24.619 INFO:teuthology.orchestra.run.smithi139.stdout: leveldb-1.22-1.el8.x86_64 2021-11-20T05:56:24.619 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.620 INFO:teuthology.orchestra.run.smithi139.stdout: liboath-2.6.2-3.el8.x86_64 2021-11-20T05:56:24.620 INFO:teuthology.orchestra.run.smithi139.stdout: libpmemobj-1.6.1-1.el8.x86_64 2021-11-20T05:56:24.620 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-11-20T05:56:24.620 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.621 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka-0.11.4-3.el8.x86_64 2021-11-20T05:56:24.621 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.621 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind-1.3.1-3.el8.x86_64 2021-11-20T05:56:24.621 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-11-20T05:56:24.621 INFO:teuthology.orchestra.run.smithi139.stdout: ndctl-libs-71.1-2.el8.x86_64 2021-11-20T05:56:24.623 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.623 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.623 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.623 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-11-20T05:56:24.624 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.624 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.624 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:24.624 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:24.624 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:56:24.714 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-test 2021-11-20T05:56:24.858 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-test 2021-11-20T05:56:25.270 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:40 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:56:25.415 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:39 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:56:26.567 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:56:26.568 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:56:26.568 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2021-11-20T05:56:26.569 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:56:26.569 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:56:26.569 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2021-11-20T05:56:26.569 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2021-11-20T05:56:26.570 INFO:teuthology.orchestra.run.smithi178.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2021-11-20T05:56:26.570 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2021-11-20T05:56:26.570 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-11-20T05:56:26.570 INFO:teuthology.orchestra.run.smithi178.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2021-11-20T05:56:26.570 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-11-20T05:56:26.571 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:26.571 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:56:26.571 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:56:26.571 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2021-11-20T05:56:26.572 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:26.572 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 53 M 2021-11-20T05:56:26.573 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 229 M 2021-11-20T05:56:26.573 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:56:26.597 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 6.3 MB/s | 169 kB 00:00 2021-11-20T05:56:26.619 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): jq-1.6-3.el8.x86_64.rpm 4.1 MB/s | 202 kB 00:00 2021-11-20T05:56:26.624 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 71 MB/s | 323 kB 00:00 2021-11-20T05:56:26.629 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 5.8 MB/s | 187 kB 00:00 2021-11-20T05:56:26.681 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:56:26.682 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:56:26.682 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-11-20T05:56:26.683 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:56:26.683 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:56:26.683 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2021-11-20T05:56:26.684 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-11-20T05:56:26.684 INFO:teuthology.orchestra.run.smithi139.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2021-11-20T05:56:26.684 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2021-11-20T05:56:26.685 INFO:teuthology.orchestra.run.smithi139.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-11-20T05:56:26.685 INFO:teuthology.orchestra.run.smithi139.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2021-11-20T05:56:26.685 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-11-20T05:56:26.686 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:26.686 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:56:26.687 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:56:26.687 INFO:teuthology.orchestra.run.smithi139.stdout:Install 6 Packages 2021-11-20T05:56:26.687 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:26.688 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 53 M 2021-11-20T05:56:26.689 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 229 M 2021-11-20T05:56:26.689 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:56:26.718 INFO:teuthology.orchestra.run.smithi139.stdout:(1/6): jq-1.6-3.el8.x86_64.rpm 5.9 MB/s | 202 kB 00:00 2021-11-20T05:56:26.721 INFO:teuthology.orchestra.run.smithi139.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 63 MB/s | 187 kB 00:00 2021-11-20T05:56:26.726 INFO:teuthology.orchestra.run.smithi139.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 68 MB/s | 323 kB 00:00 2021-11-20T05:56:26.738 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 606 kB/s | 69 kB 00:00 2021-11-20T05:56:26.744 INFO:teuthology.orchestra.run.smithi139.stdout:(4/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.8 MB/s | 169 kB 00:00 2021-11-20T05:56:26.840 INFO:teuthology.orchestra.run.smithi139.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 610 kB/s | 69 kB 00:00 2021-11-20T05:56:27.553 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 53 MB/s | 52 MB 00:00 2021-11-20T05:56:27.558 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:56:27.559 INFO:teuthology.orchestra.run.smithi178.stdout:Total 53 MB/s | 53 MB 00:00 2021-11-20T05:56:27.565 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:56:27.590 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:56:27.590 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:56:27.875 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:56:27.879 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:56:27.929 INFO:teuthology.orchestra.run.smithi139.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 42 MB/s | 52 MB 00:01 2021-11-20T05:56:27.934 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:56:27.935 INFO:teuthology.orchestra.run.smithi139.stdout:Total 42 MB/s | 53 MB 00:01 2021-11-20T05:56:27.940 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:56:27.964 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:56:27.964 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:56:28.249 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:56:28.252 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:56:28.400 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:56:28.591 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2021-11-20T05:56:28.757 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2021-11-20T05:56:28.758 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:56:28.934 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2021-11-20T05:56:28.945 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T05:56:29.029 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T05:56:29.108 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2021-11-20T05:56:29.178 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2021-11-20T05:56:29.273 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T05:56:29.346 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T05:56:29.347 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-20T05:56:29.504 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2021-11-20T05:56:29.644 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T05:56:34.143 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T05:56:34.393 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T05:56:34.460 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-20T05:56:34.651 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T05:56:34.905 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2021-11-20T05:56:34.905 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2021-11-20T05:56:34.907 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2021-11-20T05:56:34.907 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-11-20T05:56:34.907 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2021-11-20T05:56:34.908 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T05:56:34.953 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-20T05:56:34.953 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2021-11-20T05:56:34.953 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2021-11-20T05:56:34.954 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2021-11-20T05:56:34.954 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-11-20T05:56:34.954 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2021-11-20T05:56:35.101 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-11-20T05:56:35.102 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:35.103 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:56:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-3.el8.x86_64 2021-11-20T05:56:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-11-20T05:56:35.103 INFO:teuthology.orchestra.run.smithi178.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-11-20T05:56:35.104 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:35.104 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:56:35.250 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph 2021-11-20T05:56:35.362 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-11-20T05:56:35.363 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:35.363 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:56:35.363 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-3.el8.x86_64 2021-11-20T05:56:35.363 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-11-20T05:56:35.364 INFO:teuthology.orchestra.run.smithi139.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-11-20T05:56:35.364 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:35.364 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:56:35.512 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph 2021-11-20T05:56:35.789 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:50 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:56:36.056 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:50 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:56:37.093 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:56:37.096 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:56:37.096 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2021-11-20T05:56:37.097 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:56:37.097 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:56:37.097 INFO:teuthology.orchestra.run.smithi178.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2021-11-20T05:56:37.097 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2021-11-20T05:56:37.098 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2021-11-20T05:56:37.098 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2021-11-20T05:56:37.098 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2021-11-20T05:56:37.098 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2021-11-20T05:56:37.099 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2021-11-20T05:56:37.099 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-11-20T05:56:37.100 INFO:teuthology.orchestra.run.smithi178.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-11-20T05:56:37.100 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-11-20T05:56:37.100 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-11-20T05:56:37.101 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-11-20T05:56:37.101 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2021-11-20T05:56:37.101 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-11-20T05:56:37.101 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-11-20T05:56:37.102 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-11-20T05:56:37.102 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-11-20T05:56:37.103 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-11-20T05:56:37.103 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-11-20T05:56:37.103 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-11-20T05:56:37.104 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-11-20T05:56:37.104 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-11-20T05:56:37.105 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-11-20T05:56:37.105 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-11-20T05:56:37.105 INFO:teuthology.orchestra.run.smithi178.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-11-20T05:56:37.106 INFO:teuthology.orchestra.run.smithi178.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-11-20T05:56:37.106 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-11-20T05:56:37.106 INFO:teuthology.orchestra.run.smithi178.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-11-20T05:56:37.107 INFO:teuthology.orchestra.run.smithi178.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-11-20T05:56:37.107 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-11-20T05:56:37.107 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-11-20T05:56:37.108 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-11-20T05:56:37.108 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-11-20T05:56:37.109 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:37.109 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:56:37.110 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:56:37.110 INFO:teuthology.orchestra.run.smithi178.stdout:Install 32 Packages 2021-11-20T05:56:37.111 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:37.112 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 31 M 2021-11-20T05:56:37.113 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 111 M 2021-11-20T05:56:37.113 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:56:37.122 INFO:teuthology.orchestra.run.smithi178.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 317 kB/s | 6.3 kB 00:00 2021-11-20T05:56:37.163 INFO:teuthology.orchestra.run.smithi178.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 26 MB/s | 1.6 MB 00:00 2021-11-20T05:56:37.181 INFO:teuthology.orchestra.run.smithi178.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 30 MB/s | 2.4 MB 00:00 2021-11-20T05:56:37.188 INFO:teuthology.orchestra.run.smithi178.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 34 MB/s | 220 kB 00:00 2021-11-20T05:56:37.219 INFO:teuthology.orchestra.run.smithi178.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 5.0 MB/s | 157 kB 00:00 2021-11-20T05:56:37.220 INFO:teuthology.orchestra.run.smithi178.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 26 MB/s | 39 kB 00:00 2021-11-20T05:56:37.223 INFO:teuthology.orchestra.run.smithi178.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 41 MB/s | 103 kB 00:00 2021-11-20T05:56:37.225 INFO:teuthology.orchestra.run.smithi178.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 36 MB/s | 54 kB 00:00 2021-11-20T05:56:37.232 INFO:teuthology.orchestra.run.smithi178.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 68 MB/s | 457 kB 00:00 2021-11-20T05:56:37.236 INFO:teuthology.orchestra.run.smithi178.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 63 MB/s | 237 kB 00:00 2021-11-20T05:56:37.244 INFO:teuthology.orchestra.run.smithi178.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 74 MB/s | 559 kB 00:00 2021-11-20T05:56:37.248 INFO:teuthology.orchestra.run.smithi178.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 48 MB/s | 109 kB 00:00 2021-11-20T05:56:37.298 INFO:teuthology.orchestra.run.smithi178.stdout:(13/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 28 MB/s | 5.0 MB 00:00 2021-11-20T05:56:37.360 INFO:teuthology.orchestra.run.smithi178.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 384 kB/s | 44 kB 00:00 2021-11-20T05:56:37.376 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:56:37.379 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:56:37.379 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-11-20T05:56:37.379 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:56:37.380 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:56:37.380 INFO:teuthology.orchestra.run.smithi139.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2021-11-20T05:56:37.380 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-11-20T05:56:37.380 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2021-11-20T05:56:37.381 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2021-11-20T05:56:37.381 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2021-11-20T05:56:37.381 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2021-11-20T05:56:37.381 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2021-11-20T05:56:37.381 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-11-20T05:56:37.382 INFO:teuthology.orchestra.run.smithi139.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-11-20T05:56:37.382 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-11-20T05:56:37.382 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-11-20T05:56:37.382 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-11-20T05:56:37.383 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2021-11-20T05:56:37.383 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-11-20T05:56:37.383 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-11-20T05:56:37.383 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-11-20T05:56:37.383 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-11-20T05:56:37.384 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-11-20T05:56:37.384 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-11-20T05:56:37.384 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-11-20T05:56:37.384 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-11-20T05:56:37.384 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-11-20T05:56:37.385 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-11-20T05:56:37.385 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-11-20T05:56:37.385 INFO:teuthology.orchestra.run.smithi139.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-11-20T05:56:37.385 INFO:teuthology.orchestra.run.smithi139.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-11-20T05:56:37.385 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-11-20T05:56:37.386 INFO:teuthology.orchestra.run.smithi139.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-11-20T05:56:37.386 INFO:teuthology.orchestra.run.smithi139.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-11-20T05:56:37.386 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-11-20T05:56:37.386 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-11-20T05:56:37.387 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-11-20T05:56:37.387 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-11-20T05:56:37.387 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:37.387 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:56:37.387 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:56:37.388 INFO:teuthology.orchestra.run.smithi139.stdout:Install 32 Packages 2021-11-20T05:56:37.388 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:37.389 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 31 M 2021-11-20T05:56:37.390 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 111 M 2021-11-20T05:56:37.390 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:56:37.407 INFO:teuthology.orchestra.run.smithi139.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 314 kB/s | 6.3 kB 00:00 2021-11-20T05:56:37.445 INFO:teuthology.orchestra.run.smithi139.stdout:(2/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 40 MB/s | 2.4 MB 00:00 2021-11-20T05:56:37.478 INFO:teuthology.orchestra.run.smithi139.stdout:(3/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2021-11-20T05:56:37.493 INFO:teuthology.orchestra.run.smithi139.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 16 MB/s | 220 kB 00:00 2021-11-20T05:56:37.505 INFO:teuthology.orchestra.run.smithi178.stdout:(15/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 53 MB/s | 18 MB 00:00 2021-11-20T05:56:37.506 INFO:teuthology.orchestra.run.smithi178.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.2 MB/s | 173 kB 00:00 2021-11-20T05:56:37.507 INFO:teuthology.orchestra.run.smithi178.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 886 kB/s | 185 kB 00:00 2021-11-20T05:56:37.525 INFO:teuthology.orchestra.run.smithi139.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 4.8 MB/s | 157 kB 00:00 2021-11-20T05:56:37.526 INFO:teuthology.orchestra.run.smithi139.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 27 MB/s | 39 kB 00:00 2021-11-20T05:56:37.529 INFO:teuthology.orchestra.run.smithi139.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 44 MB/s | 103 kB 00:00 2021-11-20T05:56:37.530 INFO:teuthology.orchestra.run.smithi139.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 31 MB/s | 54 kB 00:00 2021-11-20T05:56:37.536 INFO:teuthology.orchestra.run.smithi178.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 387 kB/s | 11 kB 00:00 2021-11-20T05:56:37.536 INFO:teuthology.orchestra.run.smithi178.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 621 kB/s | 18 kB 00:00 2021-11-20T05:56:37.538 INFO:teuthology.orchestra.run.smithi139.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 66 MB/s | 457 kB 00:00 2021-11-20T05:56:37.542 INFO:teuthology.orchestra.run.smithi139.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 59 MB/s | 237 kB 00:00 2021-11-20T05:56:37.552 INFO:teuthology.orchestra.run.smithi139.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 64 MB/s | 559 kB 00:00 2021-11-20T05:56:37.554 INFO:teuthology.orchestra.run.smithi139.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 51 MB/s | 109 kB 00:00 2021-11-20T05:56:37.564 INFO:teuthology.orchestra.run.smithi178.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 2.0 MB/s | 59 kB 00:00 2021-11-20T05:56:37.566 INFO:teuthology.orchestra.run.smithi178.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 1.6 MB/s | 49 kB 00:00 2021-11-20T05:56:37.594 INFO:teuthology.orchestra.run.smithi178.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 545 kB/s | 16 kB 00:00 2021-11-20T05:56:37.612 INFO:teuthology.orchestra.run.smithi139.stdout:(13/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 24 MB/s | 5.0 MB 00:00 2021-11-20T05:56:37.626 INFO:teuthology.orchestra.run.smithi178.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 4.9 MB/s | 283 kB 00:00 2021-11-20T05:56:37.626 INFO:teuthology.orchestra.run.smithi178.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 658 kB/s | 19 kB 00:00 2021-11-20T05:56:37.649 INFO:teuthology.orchestra.run.smithi178.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 860 kB/s | 24 kB 00:00 2021-11-20T05:56:37.653 INFO:teuthology.orchestra.run.smithi178.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2021-11-20T05:56:37.664 INFO:teuthology.orchestra.run.smithi139.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 394 kB/s | 44 kB 00:00 2021-11-20T05:56:37.677 INFO:teuthology.orchestra.run.smithi178.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 980 kB/s | 27 kB 00:00 2021-11-20T05:56:37.705 INFO:teuthology.orchestra.run.smithi178.stdout:(28/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.9 MB/s | 384 kB 00:00 2021-11-20T05:56:37.709 INFO:teuthology.orchestra.run.smithi178.stdout:(29/32): python3-waitress-1.2.1-2.el8.1.noarch. 4.1 MB/s | 241 kB 00:00 2021-11-20T05:56:37.733 INFO:teuthology.orchestra.run.smithi178.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.4 MB/s | 251 kB 00:00 2021-11-20T05:56:37.734 INFO:teuthology.orchestra.run.smithi178.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.8 MB/s | 85 kB 00:00 2021-11-20T05:56:37.739 INFO:teuthology.orchestra.run.smithi178.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 769 kB/s | 23 kB 00:00 2021-11-20T05:56:37.743 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:56:37.745 INFO:teuthology.orchestra.run.smithi178.stdout:Total 48 MB/s | 31 MB 00:00 2021-11-20T05:56:37.760 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:56:37.773 INFO:teuthology.orchestra.run.smithi139.stdout:(15/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 55 MB/s | 18 MB 00:00 2021-11-20T05:56:37.774 INFO:teuthology.orchestra.run.smithi139.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.1 MB/s | 185 kB 00:00 2021-11-20T05:56:37.788 INFO:teuthology.orchestra.run.smithi139.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.4 MB/s | 173 kB 00:00 2021-11-20T05:56:37.797 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:56:37.797 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:56:37.801 INFO:teuthology.orchestra.run.smithi139.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 413 kB/s | 11 kB 00:00 2021-11-20T05:56:37.815 INFO:teuthology.orchestra.run.smithi139.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 632 kB/s | 18 kB 00:00 2021-11-20T05:56:37.829 INFO:teuthology.orchestra.run.smithi139.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 1.7 MB/s | 49 kB 00:00 2021-11-20T05:56:37.846 INFO:teuthology.orchestra.run.smithi139.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 2.0 MB/s | 59 kB 00:00 2021-11-20T05:56:37.872 INFO:teuthology.orchestra.run.smithi139.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 559 kB/s | 16 kB 00:00 2021-11-20T05:56:37.886 INFO:teuthology.orchestra.run.smithi139.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 4.9 MB/s | 283 kB 00:00 2021-11-20T05:56:37.900 INFO:teuthology.orchestra.run.smithi139.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 667 kB/s | 19 kB 00:00 2021-11-20T05:56:37.913 INFO:teuthology.orchestra.run.smithi139.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 886 kB/s | 24 kB 00:00 2021-11-20T05:56:37.940 INFO:teuthology.orchestra.run.smithi139.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2021-11-20T05:56:37.941 INFO:teuthology.orchestra.run.smithi139.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 1.0 MB/s | 27 kB 00:00 2021-11-20T05:56:37.967 INFO:teuthology.orchestra.run.smithi139.stdout:(28/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.9 MB/s | 384 kB 00:00 2021-11-20T05:56:37.984 INFO:teuthology.orchestra.run.smithi139.stdout:(29/32): python3-waitress-1.2.1-2.el8.1.noarch. 4.2 MB/s | 241 kB 00:00 2021-11-20T05:56:37.995 INFO:teuthology.orchestra.run.smithi139.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.9 MB/s | 85 kB 00:00 2021-11-20T05:56:37.996 INFO:teuthology.orchestra.run.smithi139.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.4 MB/s | 251 kB 00:00 2021-11-20T05:56:38.014 INFO:teuthology.orchestra.run.smithi139.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 786 kB/s | 23 kB 00:00 2021-11-20T05:56:38.017 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:56:38.017 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:56:38.018 INFO:teuthology.orchestra.run.smithi139.stdout:Total 49 MB/s | 31 MB 00:00 2021-11-20T05:56:38.020 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:56:38.031 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:56:38.073 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:56:38.073 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:56:38.300 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:56:38.300 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:56:39.144 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:56:39.368 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-11-20T05:56:39.386 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:56:39.594 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-11-20T05:56:39.617 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-11-20T05:56:39.820 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-11-20T05:56:39.851 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2021-11-20T05:56:40.042 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2021-11-20T05:56:40.076 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2021-11-20T05:56:40.164 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2021-11-20T05:56:40.290 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2021-11-20T05:56:40.354 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2021-11-20T05:56:40.473 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2021-11-20T05:56:40.545 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2021-11-20T05:56:40.663 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2021-11-20T05:56:40.725 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2021-11-20T05:56:40.878 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2021-11-20T05:56:40.924 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2021-11-20T05:56:41.074 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2021-11-20T05:56:41.145 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2021-11-20T05:56:41.308 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2021-11-20T05:56:41.365 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2021-11-20T05:56:41.557 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2021-11-20T05:56:41.588 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2021-11-20T05:56:41.795 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2021-11-20T05:56:41.992 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2021-11-20T05:56:42.035 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2021-11-20T05:56:42.288 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2021-11-20T05:56:42.440 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2021-11-20T05:56:42.498 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2021-11-20T05:56:42.687 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2021-11-20T05:56:42.740 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2021-11-20T05:56:42.908 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2021-11-20T05:56:42.961 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T05:56:43.160 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2021-11-20T05:56:43.279 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2021-11-20T05:56:43.409 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T05:56:43.508 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2021-11-20T05:56:43.689 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2021-11-20T05:56:43.720 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2021-11-20T05:56:43.924 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2021-11-20T05:56:43.924 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-11-20T05:56:44.120 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2021-11-20T05:56:44.182 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-11-20T05:56:44.322 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-11-20T05:56:44.440 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-11-20T05:56:44.572 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-11-20T05:56:44.669 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2021-11-20T05:56:44.812 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-11-20T05:56:45.047 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2021-11-20T05:56:45.074 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2021-11-20T05:56:45.386 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-11-20T05:56:45.404 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2021-11-20T05:56:45.745 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-11-20T05:56:45.789 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2021-11-20T05:56:45.976 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T05:56:46.069 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2021-11-20T05:56:46.248 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T05:56:47.496 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T05:56:47.718 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T05:56:47.769 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-20T05:56:47.981 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T05:56:48.149 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T05:56:48.375 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T05:56:48.413 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-20T05:56:48.597 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T05:56:48.673 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T05:56:48.783 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T05:56:48.800 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T05:56:48.895 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-20T05:56:48.916 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T05:56:49.099 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T05:56:49.116 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-20T05:56:49.248 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T05:56:49.296 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T05:56:49.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2021-11-20T05:56:49.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2021-11-20T05:56:49.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2021-11-20T05:56:49.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2021-11-20T05:56:49.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2021-11-20T05:56:49.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2021-11-20T05:56:49.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2021-11-20T05:56:49.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2021-11-20T05:56:49.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2021-11-20T05:56:49.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2021-11-20T05:56:49.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-11-20T05:56:49.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2021-11-20T05:56:49.300 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2021-11-20T05:56:49.300 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2021-11-20T05:56:49.300 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2021-11-20T05:56:49.300 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2021-11-20T05:56:49.300 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T05:56:49.301 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2021-11-20T05:56:49.301 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2021-11-20T05:56:49.301 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2021-11-20T05:56:49.301 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2021-11-20T05:56:49.301 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2021-11-20T05:56:49.302 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2021-11-20T05:56:49.302 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2021-11-20T05:56:49.302 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2021-11-20T05:56:49.302 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2021-11-20T05:56:49.303 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2021-11-20T05:56:49.303 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2021-11-20T05:56:49.303 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2021-11-20T05:56:49.303 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2021-11-20T05:56:49.304 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2021-11-20T05:56:49.638 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-20T05:56:49.638 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2021-11-20T05:56:49.639 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2021-11-20T05:56:49.639 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2021-11-20T05:56:49.639 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2021-11-20T05:56:49.640 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2021-11-20T05:56:49.640 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2021-11-20T05:56:49.640 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2021-11-20T05:56:49.640 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2021-11-20T05:56:49.641 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2021-11-20T05:56:49.641 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2021-11-20T05:56:49.641 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-11-20T05:56:49.641 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2021-11-20T05:56:49.642 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2021-11-20T05:56:49.642 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2021-11-20T05:56:49.642 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2021-11-20T05:56:49.642 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2021-11-20T05:56:49.643 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-20T05:56:49.643 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2021-11-20T05:56:49.643 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2021-11-20T05:56:49.643 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2021-11-20T05:56:49.644 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2021-11-20T05:56:49.644 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2021-11-20T05:56:49.644 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2021-11-20T05:56:49.644 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2021-11-20T05:56:49.645 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2021-11-20T05:56:49.645 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2021-11-20T05:56:49.645 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2021-11-20T05:56:49.645 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2021-11-20T05:56:49.648 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2021-11-20T05:56:49.648 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2021-11-20T05:56:49.648 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2021-11-20T05:56:50.129 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2021-11-20T05:56:50.129 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:50.129 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:56:50.130 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:50.130 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:50.130 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:50.130 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2021-11-20T05:56:50.131 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:50.131 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:50.131 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-11-20T05:56:50.131 INFO:teuthology.orchestra.run.smithi178.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-11-20T05:56:50.132 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-11-20T05:56:50.132 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-11-20T05:56:50.132 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-11-20T05:56:50.132 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2021-11-20T05:56:50.132 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-6.2-6.el8.noarch 2021-11-20T05:56:50.133 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-11-20T05:56:50.133 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-11-20T05:56:50.133 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako-1.0.6-13.el8.noarch 2021-11-20T05:56:50.133 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-11-20T05:56:50.133 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-11-20T05:56:50.134 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-11-20T05:56:50.134 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend-2.6-1.el8.noarch 2021-11-20T05:56:50.134 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-11-20T05:56:50.134 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser-2.14-14.el8.noarch 2021-11-20T05:56:50.135 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz-2017.2-9.el8.noarch 2021-11-20T05:56:50.135 INFO:teuthology.orchestra.run.smithi178.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-11-20T05:56:50.135 INFO:teuthology.orchestra.run.smithi178.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-11-20T05:56:50.135 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-11-20T05:56:50.135 INFO:teuthology.orchestra.run.smithi178.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-11-20T05:56:50.136 INFO:teuthology.orchestra.run.smithi178.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-11-20T05:56:50.136 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-11-20T05:56:50.137 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-11-20T05:56:50.137 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-11-20T05:56:50.137 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-11-20T05:56:50.137 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:50.137 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:56:50.331 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-base 2021-11-20T05:56:50.362 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2021-11-20T05:56:50.363 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:50.363 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:56:50.364 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:50.364 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:50.364 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:50.364 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2021-11-20T05:56:50.365 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:50.365 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2021-11-20T05:56:50.365 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-11-20T05:56:50.365 INFO:teuthology.orchestra.run.smithi139.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-11-20T05:56:50.365 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-11-20T05:56:50.366 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-11-20T05:56:50.366 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-11-20T05:56:50.366 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2021-11-20T05:56:50.366 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-6.2-6.el8.noarch 2021-11-20T05:56:50.367 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-11-20T05:56:50.367 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-11-20T05:56:50.367 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako-1.0.6-13.el8.noarch 2021-11-20T05:56:50.367 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-11-20T05:56:50.368 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-11-20T05:56:50.368 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-11-20T05:56:50.368 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend-2.6-1.el8.noarch 2021-11-20T05:56:50.368 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-11-20T05:56:50.369 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser-2.14-14.el8.noarch 2021-11-20T05:56:50.369 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz-2017.2-9.el8.noarch 2021-11-20T05:56:50.369 INFO:teuthology.orchestra.run.smithi139.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-11-20T05:56:50.369 INFO:teuthology.orchestra.run.smithi139.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-11-20T05:56:50.370 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-11-20T05:56:50.370 INFO:teuthology.orchestra.run.smithi139.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-11-20T05:56:50.370 INFO:teuthology.orchestra.run.smithi139.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-11-20T05:56:50.370 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-11-20T05:56:50.370 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-11-20T05:56:50.371 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-11-20T05:56:50.371 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-11-20T05:56:50.372 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:50.372 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:56:50.616 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-base 2021-11-20T05:56:50.872 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:05 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:56:51.160 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:05 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:56:51.866 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:56:52.148 INFO:teuthology.orchestra.run.smithi139.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:56:52.156 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:56:52.157 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2021-11-20T05:56:52.157 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:56:52.225 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install cephadm 2021-11-20T05:56:52.436 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:56:52.437 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-11-20T05:56:52.438 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:56:52.518 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install cephadm 2021-11-20T05:56:52.758 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:07 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:56:53.054 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:07 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:56:54.052 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:56:54.053 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:56:54.054 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2021-11-20T05:56:54.054 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:56:54.054 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:56:54.054 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2021-11-20T05:56:54.055 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:54.055 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:56:54.055 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:56:54.055 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2021-11-20T05:56:54.056 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:54.056 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 71 k 2021-11-20T05:56:54.056 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 290 k 2021-11-20T05:56:54.056 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:56:54.066 INFO:teuthology.orchestra.run.smithi178.stdout:cephadm-16.2.4-0.el8.noarch.rpm 6.0 MB/s | 71 kB 00:00 2021-11-20T05:56:54.066 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:56:54.067 INFO:teuthology.orchestra.run.smithi178.stdout:Total 5.4 MB/s | 71 kB 00:00 2021-11-20T05:56:54.067 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:56:54.074 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:56:54.075 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:56:54.076 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:56:54.080 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:56:54.130 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:56:54.358 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:56:54.359 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:56:54.360 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2021-11-20T05:56:54.360 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:56:54.360 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:56:54.360 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2021-11-20T05:56:54.361 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:54.361 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:56:54.361 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:56:54.361 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2021-11-20T05:56:54.362 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:54.362 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 71 k 2021-11-20T05:56:54.362 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 290 k 2021-11-20T05:56:54.363 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:56:54.372 INFO:teuthology.orchestra.run.smithi139.stdout:cephadm-16.2.4-0.el8.noarch.rpm 5.9 MB/s | 71 kB 00:00 2021-11-20T05:56:54.373 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:56:54.373 INFO:teuthology.orchestra.run.smithi139.stdout:Total 5.2 MB/s | 71 kB 00:00 2021-11-20T05:56:54.374 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:56:54.381 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:56:54.381 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:56:54.382 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:56:54.386 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:56:54.431 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:56:56.700 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:56:56.987 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:56:57.024 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:56:57.185 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:56:57.301 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:56:57.499 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:56:57.531 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:56:57.531 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:57.531 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:56:57.531 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm-2:16.2.4-0.el8.noarch 2021-11-20T05:56:57.532 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:57.532 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:56:57.704 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-immutable-object-cache 2021-11-20T05:56:57.810 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-20T05:56:57.810 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:57.811 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:56:57.811 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm-2:16.2.4-0.el8.noarch 2021-11-20T05:56:57.811 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:57.812 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:56:57.969 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-immutable-object-cache 2021-11-20T05:56:58.263 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:13 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:56:58.521 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:12 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:56:59.543 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:56:59.545 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:56:59.545 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2021-11-20T05:56:59.546 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:56:59.546 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:56:59.547 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2021-11-20T05:56:59.547 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:59.547 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:56:59.548 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:56:59.548 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2021-11-20T05:56:59.549 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:56:59.549 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 163 k 2021-11-20T05:56:59.549 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 587 k 2021-11-20T05:56:59.550 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:56:59.559 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 12 MB/s | 163 kB 00:00 2021-11-20T05:56:59.559 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:56:59.559 INFO:teuthology.orchestra.run.smithi178.stdout:Total 11 MB/s | 163 kB 00:00 2021-11-20T05:56:59.559 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:56:59.572 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:56:59.572 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:56:59.619 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:56:59.622 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:56:59.736 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:56:59.812 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:56:59.812 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:56:59.812 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2021-11-20T05:56:59.813 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:56:59.813 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:56:59.813 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2021-11-20T05:56:59.813 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:59.814 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:56:59.814 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:56:59.814 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2021-11-20T05:56:59.816 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:56:59.816 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 163 k 2021-11-20T05:56:59.816 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 587 k 2021-11-20T05:56:59.817 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:56:59.826 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 12 MB/s | 163 kB 00:00 2021-11-20T05:56:59.827 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:56:59.827 INFO:teuthology.orchestra.run.smithi139.stdout:Total 10 MB/s | 163 kB 00:00 2021-11-20T05:56:59.828 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:56:59.840 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:56:59.841 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:56:59.889 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:56:59.892 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:56:59.973 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:00.006 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:57:00.222 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:00.365 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:00.632 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:00.680 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:00.680 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:00.681 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:57:00.681 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2021-11-20T05:57:00.681 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:00.681 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:00.884 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr 2021-11-20T05:57:00.958 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:00.958 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:00.959 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:57:00.959 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2021-11-20T05:57:00.959 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:00.959 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:57:01.131 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr 2021-11-20T05:57:01.436 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:16 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:57:01.683 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:15 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:57:03.048 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:57:03.287 INFO:teuthology.orchestra.run.smithi139.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:57:03.335 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:57:03.336 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2021-11-20T05:57:03.336 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:03.408 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-dashboard 2021-11-20T05:57:03.574 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:57:03.575 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-11-20T05:57:03.575 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:57:03.648 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-dashboard 2021-11-20T05:57:03.947 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:18 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:57:04.186 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:18 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:57:05.280 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:57:05.281 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:05.282 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2021-11-20T05:57:05.282 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:05.282 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:57:05.283 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2021-11-20T05:57:05.283 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2021-11-20T05:57:05.283 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2021-11-20T05:57:05.283 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2021-11-20T05:57:05.284 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-11-20T05:57:05.284 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-11-20T05:57:05.284 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-11-20T05:57:05.284 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:05.284 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:57:05.285 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:05.285 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2021-11-20T05:57:05.285 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:05.285 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 3.0 M 2021-11-20T05:57:05.286 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 15 M 2021-11-20T05:57:05.287 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:57:05.304 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 1.2 MB/s | 24 kB 00:00 2021-11-20T05:57:05.306 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 404 kB/s | 8.9 kB 00:00 2021-11-20T05:57:05.334 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 52 MB/s | 2.7 MB 00:00 2021-11-20T05:57:05.452 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:57:05.453 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-11-20T05:57:05.453 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:05.453 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:57:05.453 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2021-11-20T05:57:05.454 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-11-20T05:57:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2021-11-20T05:57:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2021-11-20T05:57:05.454 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-11-20T05:57:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-11-20T05:57:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-11-20T05:57:05.455 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:05.456 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:57:05.456 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:05.456 INFO:teuthology.orchestra.run.smithi139.stdout:Install 6 Packages 2021-11-20T05:57:05.456 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:05.457 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 3.0 M 2021-11-20T05:57:05.457 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 15 M 2021-11-20T05:57:05.457 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:57:05.462 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 215 kB/s | 34 kB 00:00 2021-11-20T05:57:05.475 INFO:teuthology.orchestra.run.smithi139.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 1.2 MB/s | 24 kB 00:00 2021-11-20T05:57:05.476 INFO:teuthology.orchestra.run.smithi139.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 430 kB/s | 8.9 kB 00:00 2021-11-20T05:57:05.505 INFO:teuthology.orchestra.run.smithi139.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 53 MB/s | 2.7 MB 00:00 2021-11-20T05:57:05.558 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 886 kB/s | 196 kB 00:00 2021-11-20T05:57:05.608 INFO:teuthology.orchestra.run.smithi139.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 256 kB/s | 34 kB 00:00 2021-11-20T05:57:06.078 INFO:teuthology.orchestra.run.smithi139.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 343 kB/s | 196 kB 00:00 2021-11-20T05:57:08.097 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 15 kB/s | 43 kB 00:02 2021-11-20T05:57:08.101 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:57:08.102 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.0 MB/s | 3.0 MB 00:02 2021-11-20T05:57:08.104 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:57:08.115 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:57:08.118 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:57:08.313 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:57:08.318 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:57:08.820 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:57:09.096 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-11-20T05:57:09.267 INFO:teuthology.orchestra.run.smithi139.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 11 kB/s | 43 kB 00:03 2021-11-20T05:57:09.271 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:57:09.272 INFO:teuthology.orchestra.run.smithi139.stdout:Total 791 kB/s | 3.0 MB 00:03 2021-11-20T05:57:09.275 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:57:09.285 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:57:09.287 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:57:09.316 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-11-20T05:57:09.365 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:57:09.368 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:57:09.531 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-11-20T05:57:09.651 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:57:09.669 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2021-11-20T05:57:09.920 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-11-20T05:57:10.139 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-11-20T05:57:10.330 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-11-20T05:57:10.494 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2021-11-20T05:57:10.749 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2021-11-20T05:57:11.012 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:57:11.255 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:57:11.255 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2021-11-20T05:57:11.256 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2021-11-20T05:57:11.256 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2021-11-20T05:57:11.256 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-11-20T05:57:11.256 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-11-20T05:57:11.546 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2021-11-20T05:57:11.589 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-11-20T05:57:11.590 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:11.590 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:57:11.590 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2021-11-20T05:57:11.591 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2021-11-20T05:57:11.591 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2021-11-20T05:57:11.592 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-11-20T05:57:11.592 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-11-20T05:57:11.593 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes-2.4.1-12.el8.noarch 2021-11-20T05:57:11.593 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:11.594 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:11.698 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-diskprediction-local 2021-11-20T05:57:11.827 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:57:12.078 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:57:12.079 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2021-11-20T05:57:12.079 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2021-11-20T05:57:12.079 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2021-11-20T05:57:12.080 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-11-20T05:57:12.080 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-11-20T05:57:12.240 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:27 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:57:12.389 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-11-20T05:57:12.390 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:12.390 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:57:12.390 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2021-11-20T05:57:12.391 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2021-11-20T05:57:12.391 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2021-11-20T05:57:12.391 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-11-20T05:57:12.391 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-11-20T05:57:12.391 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes-2.4.1-12.el8.noarch 2021-11-20T05:57:12.392 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:12.392 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:57:12.475 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-diskprediction-local 2021-11-20T05:57:13.012 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:27 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:57:13.520 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:57:13.522 INFO:teuthology.orchestra.run.smithi178.stdout:===================================================================================================== 2021-11-20T05:57:13.522 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2021-11-20T05:57:13.523 INFO:teuthology.orchestra.run.smithi178.stdout:===================================================================================================== 2021-11-20T05:57:13.523 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:57:13.523 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2021-11-20T05:57:13.524 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2021-11-20T05:57:13.524 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran x86_64 8.5.0-3.el8 CentOS-Base 643 k 2021-11-20T05:57:13.524 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath x86_64 8.5.0-3.el8 CentOS-Base 170 k 2021-11-20T05:57:13.524 INFO:teuthology.orchestra.run.smithi178.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2021-11-20T05:57:13.524 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2021-11-20T05:57:13.525 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-11-20T05:57:13.525 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-11-20T05:57:13.526 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2021-11-20T05:57:13.526 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:13.526 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:57:13.526 INFO:teuthology.orchestra.run.smithi178.stdout:===================================================================================================== 2021-11-20T05:57:13.527 INFO:teuthology.orchestra.run.smithi178.stdout:Install 8 Packages 2021-11-20T05:57:13.527 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:13.528 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 37 M 2021-11-20T05:57:13.528 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 211 M 2021-11-20T05:57:13.529 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:57:13.643 INFO:teuthology.orchestra.run.smithi178.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 73 MB/s | 8.6 MB 00:00 2021-11-20T05:57:13.672 INFO:teuthology.orchestra.run.smithi178.stdout:(2/8): openblas-0.3.15-3.el8.x86_64.rpm 32 MB/s | 4.7 MB 00:00 2021-11-20T05:57:13.679 INFO:teuthology.orchestra.run.smithi178.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 32 MB/s | 225 kB 00:00 2021-11-20T05:57:13.695 INFO:teuthology.orchestra.run.smithi178.stdout:(4/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 29 MB/s | 4.9 MB 00:00 2021-11-20T05:57:13.712 INFO:teuthology.orchestra.run.smithi178.stdout:(5/8): libgfortran-8.5.0-3.el8.x86_64.rpm 38 MB/s | 643 kB 00:00 2021-11-20T05:57:13.718 INFO:teuthology.orchestra.run.smithi178.stdout:(6/8): libquadmath-8.5.0-3.el8.x86_64.rpm 27 MB/s | 170 kB 00:00 2021-11-20T05:57:13.951 INFO:teuthology.orchestra.run.smithi178.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 51 MB/s | 14 MB 00:00 2021-11-20T05:57:13.980 INFO:teuthology.orchestra.run.smithi178.stdout:(8/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2021-11-20T05:57:13.985 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:57:13.985 INFO:teuthology.orchestra.run.smithi178.stdout:Total 79 MB/s | 37 MB 00:00 2021-11-20T05:57:13.999 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:57:14.019 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:57:14.019 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:57:14.314 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:57:14.315 INFO:teuthology.orchestra.run.smithi139.stdout:===================================================================================================== 2021-11-20T05:57:14.315 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-11-20T05:57:14.315 INFO:teuthology.orchestra.run.smithi139.stdout:===================================================================================================== 2021-11-20T05:57:14.315 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:57:14.316 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2021-11-20T05:57:14.316 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-11-20T05:57:14.316 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran x86_64 8.5.0-3.el8 CentOS-Base 643 k 2021-11-20T05:57:14.316 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath x86_64 8.5.0-3.el8 CentOS-Base 170 k 2021-11-20T05:57:14.317 INFO:teuthology.orchestra.run.smithi139.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2021-11-20T05:57:14.317 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2021-11-20T05:57:14.317 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-11-20T05:57:14.317 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-11-20T05:57:14.318 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2021-11-20T05:57:14.318 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:14.318 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:57:14.318 INFO:teuthology.orchestra.run.smithi139.stdout:===================================================================================================== 2021-11-20T05:57:14.319 INFO:teuthology.orchestra.run.smithi139.stdout:Install 8 Packages 2021-11-20T05:57:14.319 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:14.320 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:57:14.320 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 37 M 2021-11-20T05:57:14.320 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 211 M 2021-11-20T05:57:14.321 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:57:14.323 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:57:14.420 INFO:teuthology.orchestra.run.smithi139.stdout:(1/8): openblas-0.3.15-3.el8.x86_64.rpm 46 MB/s | 4.7 MB 00:00 2021-11-20T05:57:14.447 INFO:teuthology.orchestra.run.smithi139.stdout:(2/8): ceph-mgr-diskprediction-local-16.2.4-0.e 67 MB/s | 8.6 MB 00:00 2021-11-20T05:57:14.499 INFO:teuthology.orchestra.run.smithi139.stdout:(3/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 48 MB/s | 3.7 MB 00:00 2021-11-20T05:57:14.514 INFO:teuthology.orchestra.run.smithi139.stdout:(4/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 3.5 MB/s | 225 kB 00:00 2021-11-20T05:57:14.577 INFO:teuthology.orchestra.run.smithi139.stdout:(5/8): libgfortran-8.5.0-3.el8.x86_64.rpm 9.4 MB/s | 643 kB 00:00 2021-11-20T05:57:14.590 INFO:teuthology.orchestra.run.smithi139.stdout:(6/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.9 MB 00:00 2021-11-20T05:57:14.592 INFO:teuthology.orchestra.run.smithi139.stdout:(7/8): libquadmath-8.5.0-3.el8.x86_64.rpm 11 MB/s | 170 kB 00:00 2021-11-20T05:57:14.782 INFO:teuthology.orchestra.run.smithi139.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 49 MB/s | 14 MB 00:00 2021-11-20T05:57:14.785 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:57:14.786 INFO:teuthology.orchestra.run.smithi139.stdout:Total 78 MB/s | 37 MB 00:00 2021-11-20T05:57:14.800 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:57:14.819 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:57:14.819 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:57:14.857 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:57:14.986 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T05:57:15.121 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:57:15.123 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T05:57:15.125 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:57:15.279 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T05:57:15.657 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:57:15.802 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T05:57:15.870 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T05:57:15.959 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-20T05:57:16.078 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T05:57:16.119 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T05:57:16.689 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T05:57:16.724 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-20T05:57:16.852 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T05:57:16.929 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T05:57:17.542 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2021-11-20T05:57:17.587 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T05:57:17.743 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T05:57:17.936 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T05:57:18.169 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T05:57:18.472 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-20T05:57:18.837 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T05:57:19.084 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T05:57:20.078 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T05:57:20.919 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-20T05:57:21.394 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2021-11-20T05:57:21.594 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T05:57:21.888 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T05:57:21.889 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2021-11-20T05:57:21.889 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2021-11-20T05:57:21.889 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2021-11-20T05:57:21.889 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-11-20T05:57:21.890 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T05:57:21.890 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2021-11-20T05:57:21.890 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libgfortran-8.5.0-3.el8.x86_64 7/8 2021-11-20T05:57:22.244 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2021-11-20T05:57:22.245 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libquadmath-8.5.0-3.el8.x86_64 8/8 2021-11-20T05:57:22.246 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:22.246 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:57:22.246 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2021-11-20T05:57:22.246 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran-8.5.0-3.el8.x86_64 2021-11-20T05:57:22.247 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath-8.5.0-3.el8.x86_64 2021-11-20T05:57:22.247 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-0.3.15-3.el8.x86_64 2021-11-20T05:57:22.247 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-threads-0.3.15-3.el8.x86_64 2021-11-20T05:57:22.247 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-11-20T05:57:22.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-11-20T05:57:22.248 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2021-11-20T05:57:22.248 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:22.248 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:22.358 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-rook 2021-11-20T05:57:22.443 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T05:57:22.741 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-20T05:57:22.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2021-11-20T05:57:22.741 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2021-11-20T05:57:22.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2021-11-20T05:57:22.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-11-20T05:57:22.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-11-20T05:57:22.742 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2021-11-20T05:57:22.743 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libgfortran-8.5.0-3.el8.x86_64 7/8 2021-11-20T05:57:22.897 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:37 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:57:23.072 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libquadmath-8.5.0-3.el8.x86_64 8/8 2021-11-20T05:57:23.072 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:23.072 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:57:23.072 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2021-11-20T05:57:23.073 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran-8.5.0-3.el8.x86_64 2021-11-20T05:57:23.073 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath-8.5.0-3.el8.x86_64 2021-11-20T05:57:23.073 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-0.3.15-3.el8.x86_64 2021-11-20T05:57:23.073 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-threads-0.3.15-3.el8.x86_64 2021-11-20T05:57:23.074 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-11-20T05:57:23.074 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-11-20T05:57:23.074 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2021-11-20T05:57:23.074 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:23.074 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:57:23.170 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-rook 2021-11-20T05:57:23.709 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:37 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:57:24.183 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:57:24.184 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:24.185 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2021-11-20T05:57:24.185 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:24.185 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:57:24.185 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2021-11-20T05:57:24.185 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2021-11-20T05:57:24.186 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-11-20T05:57:24.186 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-11-20T05:57:24.186 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-11-20T05:57:24.186 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-11-20T05:57:24.187 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-11-20T05:57:24.187 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-11-20T05:57:24.187 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-11-20T05:57:24.187 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-11-20T05:57:24.187 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-11-20T05:57:24.188 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-11-20T05:57:24.188 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa noarch 4.7.2-1.el8 epel 61 k 2021-11-20T05:57:24.188 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-11-20T05:57:24.188 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:24.189 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:57:24.189 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:24.189 INFO:teuthology.orchestra.run.smithi178.stdout:Install 13 Packages 2021-11-20T05:57:24.189 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:24.190 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 1.9 M 2021-11-20T05:57:24.190 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 24 M 2021-11-20T05:57:24.190 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:57:24.207 INFO:teuthology.orchestra.run.smithi178.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 27 kB 00:00 2021-11-20T05:57:24.223 INFO:teuthology.orchestra.run.smithi178.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 750 kB/s | 27 kB 00:00 2021-11-20T05:57:24.226 INFO:teuthology.orchestra.run.smithi178.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 46 MB/s | 110 kB 00:00 2021-11-20T05:57:24.228 INFO:teuthology.orchestra.run.smithi178.stdout:(4/13): python3-jsonpointer-1.10-11.el8.noarch. 502 kB/s | 20 kB 00:00 2021-11-20T05:57:24.228 INFO:teuthology.orchestra.run.smithi178.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 58 MB/s | 155 kB 00:00 2021-11-20T05:57:24.229 INFO:teuthology.orchestra.run.smithi178.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 29 MB/s | 43 kB 00:00 2021-11-20T05:57:24.238 INFO:teuthology.orchestra.run.smithi178.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.9 MB/s | 126 kB 00:00 2021-11-20T05:57:24.316 INFO:teuthology.orchestra.run.smithi178.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 196 kB/s | 16 kB 00:00 2021-11-20T05:57:24.316 INFO:teuthology.orchestra.run.smithi178.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 392 kB/s | 33 kB 00:00 2021-11-20T05:57:24.349 INFO:teuthology.orchestra.run.smithi178.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 736 kB/s | 81 kB 00:00 2021-11-20T05:57:24.369 INFO:teuthology.orchestra.run.smithi178.stdout:(11/13): python3-rsa-4.7.2-1.el8.noarch.rpm 1.1 MB/s | 61 kB 00:00 2021-11-20T05:57:24.378 INFO:teuthology.orchestra.run.smithi178.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 2.1 MB/s | 61 kB 00:00 2021-11-20T05:57:24.508 INFO:teuthology.orchestra.run.smithi178.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 6.0 MB/s | 1.2 MB 00:00 2021-11-20T05:57:24.511 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:57:24.511 INFO:teuthology.orchestra.run.smithi178.stdout:Total 5.9 MB/s | 1.9 MB 00:00 2021-11-20T05:57:24.523 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:57:24.536 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:57:24.536 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:57:24.612 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:57:24.615 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:57:25.050 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:57:25.051 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:25.051 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-11-20T05:57:25.051 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:25.051 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:57:25.052 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2021-11-20T05:57:25.052 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-11-20T05:57:25.052 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-11-20T05:57:25.052 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-11-20T05:57:25.052 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-11-20T05:57:25.053 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-11-20T05:57:25.053 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-11-20T05:57:25.053 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-11-20T05:57:25.053 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-11-20T05:57:25.054 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-11-20T05:57:25.054 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-11-20T05:57:25.054 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-11-20T05:57:25.054 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa noarch 4.7.2-1.el8 epel 61 k 2021-11-20T05:57:25.054 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-11-20T05:57:25.055 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:25.055 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:57:25.055 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:25.055 INFO:teuthology.orchestra.run.smithi139.stdout:Install 13 Packages 2021-11-20T05:57:25.056 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:25.056 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 1.9 M 2021-11-20T05:57:25.056 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 24 M 2021-11-20T05:57:25.056 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:57:25.076 INFO:teuthology.orchestra.run.smithi139.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 27 kB 00:00 2021-11-20T05:57:25.091 INFO:teuthology.orchestra.run.smithi139.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 754 kB/s | 27 kB 00:00 2021-11-20T05:57:25.091 INFO:teuthology.orchestra.run.smithi139.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 42 MB/s | 110 kB 00:00 2021-11-20T05:57:25.091 INFO:teuthology.orchestra.run.smithi139.stdout:(4/13): python3-jsonpointer-1.10-11.el8.noarch. 508 kB/s | 20 kB 00:00 2021-11-20T05:57:25.111 INFO:teuthology.orchestra.run.smithi139.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 49 MB/s | 155 kB 00:00 2021-11-20T05:57:25.112 INFO:teuthology.orchestra.run.smithi139.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 18 MB/s | 43 kB 00:00 2021-11-20T05:57:25.112 INFO:teuthology.orchestra.run.smithi139.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.1 MB/s | 126 kB 00:00 2021-11-20T05:57:25.122 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:57:25.178 INFO:teuthology.orchestra.run.smithi139.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 197 kB/s | 16 kB 00:00 2021-11-20T05:57:25.178 INFO:teuthology.orchestra.run.smithi139.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 394 kB/s | 33 kB 00:00 2021-11-20T05:57:25.214 INFO:teuthology.orchestra.run.smithi139.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 754 kB/s | 81 kB 00:00 2021-11-20T05:57:25.232 INFO:teuthology.orchestra.run.smithi139.stdout:(11/13): python3-rsa-4.7.2-1.el8.noarch.rpm 1.1 MB/s | 61 kB 00:00 2021-11-20T05:57:25.236 INFO:teuthology.orchestra.run.smithi139.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 2.2 MB/s | 61 kB 00:00 2021-11-20T05:57:25.332 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-11-20T05:57:25.375 INFO:teuthology.orchestra.run.smithi139.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.9 MB/s | 1.2 MB 00:00 2021-11-20T05:57:25.376 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:57:25.376 INFO:teuthology.orchestra.run.smithi139.stdout:Total 5.8 MB/s | 1.9 MB 00:00 2021-11-20T05:57:25.386 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:57:25.400 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:57:25.401 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:57:25.477 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:57:25.480 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:57:25.547 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-11-20T05:57:25.781 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rsa-4.7.2-1.el8.noarch 3/13 2021-11-20T05:57:25.952 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:57:26.025 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-11-20T05:57:26.164 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-11-20T05:57:26.244 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-11-20T05:57:26.338 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-11-20T05:57:26.497 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-11-20T05:57:26.554 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rsa-4.7.2-1.el8.noarch 3/13 2021-11-20T05:57:26.751 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-11-20T05:57:26.802 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-11-20T05:57:26.999 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-11-20T05:57:27.007 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-11-20T05:57:27.249 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-11-20T05:57:27.527 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-11-20T05:57:27.814 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-11-20T05:57:28.110 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-11-20T05:57:28.391 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-11-20T05:57:28.550 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-11-20T05:57:28.733 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-11-20T05:57:28.858 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T05:57:28.912 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-11-20T05:57:29.116 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T05:57:29.116 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2021-11-20T05:57:29.117 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-11-20T05:57:29.117 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-11-20T05:57:29.117 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-11-20T05:57:29.117 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-11-20T05:57:29.118 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-11-20T05:57:29.118 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-11-20T05:57:29.118 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-11-20T05:57:29.118 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-11-20T05:57:29.118 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-11-20T05:57:29.119 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-11-20T05:57:29.119 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rsa-4.7.2-1.el8.noarch 12/13 2021-11-20T05:57:29.166 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-11-20T05:57:29.332 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-11-20T05:57:29.527 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-11-20T05:57:29.528 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:29.528 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:57:29.528 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2021-11-20T05:57:29.528 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-11-20T05:57:29.529 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-11-20T05:57:29.529 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-11-20T05:57:29.529 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-11-20T05:57:29.529 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-11-20T05:57:29.530 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-11-20T05:57:29.530 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-11-20T05:57:29.530 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-11-20T05:57:29.530 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-11-20T05:57:29.531 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-11-20T05:57:29.531 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa-4.7.2-1.el8.noarch 2021-11-20T05:57:29.531 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-11-20T05:57:29.531 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:29.531 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:29.549 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-11-20T05:57:29.628 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-cephadm 2021-11-20T05:57:29.708 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T05:57:29.970 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-20T05:57:29.970 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2021-11-20T05:57:29.971 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-11-20T05:57:29.971 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-11-20T05:57:29.971 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-11-20T05:57:29.971 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-11-20T05:57:29.971 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-11-20T05:57:29.972 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-11-20T05:57:29.972 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-11-20T05:57:29.972 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-11-20T05:57:29.972 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-11-20T05:57:29.973 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-11-20T05:57:29.973 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rsa-4.7.2-1.el8.noarch 12/13 2021-11-20T05:57:30.167 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:45 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:57:30.353 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-11-20T05:57:30.354 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:30.354 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:57:30.355 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2021-11-20T05:57:30.355 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-11-20T05:57:30.355 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-11-20T05:57:30.355 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-11-20T05:57:30.356 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-11-20T05:57:30.356 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-11-20T05:57:30.356 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-11-20T05:57:30.356 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-11-20T05:57:30.356 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-11-20T05:57:30.357 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-11-20T05:57:30.357 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-11-20T05:57:30.357 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa-4.7.2-1.el8.noarch 2021-11-20T05:57:30.357 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-11-20T05:57:30.358 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:30.358 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:57:30.817 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-cephadm 2021-11-20T05:57:31.361 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:45 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:57:31.487 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:57:31.487 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:31.488 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2021-11-20T05:57:31.488 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:31.489 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:57:31.489 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2021-11-20T05:57:31.489 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2021-11-20T05:57:31.489 INFO:teuthology.orchestra.run.smithi178.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-11-20T05:57:31.489 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2021-11-20T05:57:31.490 INFO:teuthology.orchestra.run.smithi178.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-11-20T05:57:31.490 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2021-11-20T05:57:31.490 INFO:teuthology.orchestra.run.smithi178.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2021-11-20T05:57:31.490 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:31.491 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:57:31.491 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:31.491 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2021-11-20T05:57:31.492 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:31.494 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 5.8 M 2021-11-20T05:57:31.494 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 24 M 2021-11-20T05:57:31.494 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:57:31.510 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 4.5 MB/s | 89 kB 00:00 2021-11-20T05:57:31.549 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 8.9 MB/s | 538 kB 00:00 2021-11-20T05:57:31.589 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 48 MB/s | 4.8 MB 00:00 2021-11-20T05:57:31.593 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 252 kB/s | 21 kB 00:00 2021-11-20T05:57:31.700 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 442 kB/s | 49 kB 00:00 2021-11-20T05:57:31.730 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.9 MB/s | 355 kB 00:00 2021-11-20T05:57:31.731 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:57:31.732 INFO:teuthology.orchestra.run.smithi178.stdout:Total 24 MB/s | 5.8 MB 00:00 2021-11-20T05:57:31.736 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:57:31.746 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:57:31.746 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:57:31.830 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:57:31.833 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:57:32.098 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:57:32.379 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-11-20T05:57:32.638 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-11-20T05:57:32.648 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:57:32.649 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:32.650 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2021-11-20T05:57:32.650 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:32.650 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:57:32.650 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2021-11-20T05:57:32.651 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-11-20T05:57:32.651 INFO:teuthology.orchestra.run.smithi139.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-11-20T05:57:32.651 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2021-11-20T05:57:32.651 INFO:teuthology.orchestra.run.smithi139.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-11-20T05:57:32.652 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2021-11-20T05:57:32.652 INFO:teuthology.orchestra.run.smithi139.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2021-11-20T05:57:32.652 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:32.652 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:57:32.653 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:32.653 INFO:teuthology.orchestra.run.smithi139.stdout:Install 6 Packages 2021-11-20T05:57:32.653 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:32.654 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 5.8 M 2021-11-20T05:57:32.654 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 24 M 2021-11-20T05:57:32.654 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:57:32.671 INFO:teuthology.orchestra.run.smithi139.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 4.5 MB/s | 89 kB 00:00 2021-11-20T05:57:32.707 INFO:teuthology.orchestra.run.smithi139.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 9.6 MB/s | 538 kB 00:00 2021-11-20T05:57:32.752 INFO:teuthology.orchestra.run.smithi139.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 48 MB/s | 4.8 MB 00:00 2021-11-20T05:57:32.755 INFO:teuthology.orchestra.run.smithi139.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 253 kB/s | 21 kB 00:00 2021-11-20T05:57:32.863 INFO:teuthology.orchestra.run.smithi139.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 444 kB/s | 49 kB 00:00 2021-11-20T05:57:32.872 INFO:teuthology.orchestra.run.smithi139.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.1 MB/s | 355 kB 00:00 2021-11-20T05:57:32.874 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:57:32.875 INFO:teuthology.orchestra.run.smithi139.stdout:Total 26 MB/s | 5.8 MB 00:00 2021-11-20T05:57:32.879 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:57:32.889 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:57:32.889 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:57:32.979 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:57:32.980 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:57:33.263 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:57:33.504 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-11-20T05:57:33.585 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2021-11-20T05:57:33.762 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-11-20T05:57:33.947 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2021-11-20T05:57:34.181 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2021-11-20T05:57:34.382 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:57:34.639 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:57:34.640 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2021-11-20T05:57:34.640 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2021-11-20T05:57:34.641 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2021-11-20T05:57:34.641 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-11-20T05:57:34.641 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-11-20T05:57:34.728 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2021-11-20T05:57:34.960 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2021-11-20T05:57:34.960 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:34.961 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:57:34.963 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2021-11-20T05:57:34.964 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2021-11-20T05:57:34.964 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2021-11-20T05:57:34.964 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:34.965 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:35.062 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-fuse 2021-11-20T05:57:35.137 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2021-11-20T05:57:35.388 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2021-11-20T05:57:35.583 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:57:35.603 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:50 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:57:35.835 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-20T05:57:35.835 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2021-11-20T05:57:35.836 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2021-11-20T05:57:35.836 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2021-11-20T05:57:35.836 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-11-20T05:57:35.836 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-11-20T05:57:36.160 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2021-11-20T05:57:36.161 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:36.161 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:57:36.161 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2021-11-20T05:57:36.162 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2021-11-20T05:57:36.162 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2021-11-20T05:57:36.162 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:36.162 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:57:36.265 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-fuse 2021-11-20T05:57:36.809 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:50 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:57:36.896 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:57:36.897 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:36.897 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2021-11-20T05:57:36.898 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:36.898 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:57:36.898 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2021-11-20T05:57:36.898 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2021-11-20T05:57:36.899 INFO:teuthology.orchestra.run.smithi178.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-11-20T05:57:36.899 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:36.899 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:57:36.899 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:36.900 INFO:teuthology.orchestra.run.smithi178.stdout:Install 2 Packages 2021-11-20T05:57:36.900 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:36.900 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 967 k 2021-11-20T05:57:36.900 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 3.2 M 2021-11-20T05:57:36.901 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:57:36.923 INFO:teuthology.orchestra.run.smithi178.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 36 MB/s | 885 kB 00:00 2021-11-20T05:57:36.935 INFO:teuthology.orchestra.run.smithi178.stdout:(2/2): fuse-2.9.7-12.el8.x86_64.rpm 2.2 MB/s | 82 kB 00:00 2021-11-20T05:57:36.936 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:57:36.937 INFO:teuthology.orchestra.run.smithi178.stdout:Total 24 MB/s | 967 kB 00:00 2021-11-20T05:57:36.939 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:57:36.960 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:57:36.961 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:57:37.043 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:57:37.046 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:57:37.207 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:57:37.532 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-11-20T05:57:37.732 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T05:57:38.101 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T05:57:38.101 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2021-11-20T05:57:38.148 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:57:38.149 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:38.150 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2021-11-20T05:57:38.150 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:38.150 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:57:38.150 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2021-11-20T05:57:38.151 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2021-11-20T05:57:38.151 INFO:teuthology.orchestra.run.smithi139.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-11-20T05:57:38.151 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:38.151 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:57:38.152 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:38.152 INFO:teuthology.orchestra.run.smithi139.stdout:Install 2 Packages 2021-11-20T05:57:38.152 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:38.152 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 967 k 2021-11-20T05:57:38.152 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 3.2 M 2021-11-20T05:57:38.153 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:57:38.164 INFO:teuthology.orchestra.run.smithi139.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 36 MB/s | 885 kB 00:00 2021-11-20T05:57:38.177 INFO:teuthology.orchestra.run.smithi139.stdout:(2/2): fuse-2.9.7-12.el8.x86_64.rpm 2.2 MB/s | 82 kB 00:00 2021-11-20T05:57:38.179 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:57:38.179 INFO:teuthology.orchestra.run.smithi139.stdout:Total 24 MB/s | 967 kB 00:00 2021-11-20T05:57:38.181 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:57:38.203 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:57:38.203 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:57:38.286 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:57:38.289 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:57:38.440 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-11-20T05:57:38.440 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:38.440 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:57:38.441 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-11-20T05:57:38.441 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:38.441 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:38.452 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:57:38.615 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados-devel 2021-11-20T05:57:38.765 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-11-20T05:57:38.973 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T05:57:39.163 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:54 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:57:39.346 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-20T05:57:39.347 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2021-11-20T05:57:39.678 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-11-20T05:57:39.678 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:39.678 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:57:39.679 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-11-20T05:57:39.679 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:39.679 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:57:39.853 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librados-devel 2021-11-20T05:57:40.408 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:54 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:57:40.486 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:57:40.486 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:40.487 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2021-11-20T05:57:40.487 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:40.487 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:57:40.487 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2021-11-20T05:57:40.487 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:40.488 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:57:40.488 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:40.488 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2021-11-20T05:57:40.488 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:40.489 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 131 k 2021-11-20T05:57:40.489 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 439 k 2021-11-20T05:57:40.489 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:57:40.499 INFO:teuthology.orchestra.run.smithi178.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 10 MB/s | 131 kB 00:00 2021-11-20T05:57:40.500 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:57:40.500 INFO:teuthology.orchestra.run.smithi178.stdout:Total 9.4 MB/s | 131 kB 00:00 2021-11-20T05:57:40.500 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:57:40.507 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:57:40.508 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:57:40.558 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:57:40.561 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:57:40.679 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:57:40.881 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:41.324 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:41.653 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:41.653 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:41.653 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:57:41.654 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2021-11-20T05:57:41.654 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:41.654 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:41.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs2 2021-11-20T05:57:41.867 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:57:41.868 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:41.868 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2021-11-20T05:57:41.868 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:41.868 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:57:41.869 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2021-11-20T05:57:41.869 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:41.869 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:57:41.869 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:41.870 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2021-11-20T05:57:41.870 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:41.870 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 131 k 2021-11-20T05:57:41.870 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 439 k 2021-11-20T05:57:41.870 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:57:41.886 INFO:teuthology.orchestra.run.smithi139.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 7.5 MB/s | 131 kB 00:00 2021-11-20T05:57:41.886 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:57:41.887 INFO:teuthology.orchestra.run.smithi139.stdout:Total 6.9 MB/s | 131 kB 00:00 2021-11-20T05:57:41.887 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:57:41.894 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:57:41.894 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:57:41.945 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:57:41.949 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:57:42.058 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:57:42.296 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:42.380 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:57 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:57:42.559 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:42.896 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:42.897 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:42.897 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:57:42.897 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2021-11-20T05:57:42.897 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:42.898 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:57:43.105 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install libcephfs2 2021-11-20T05:57:43.406 INFO:teuthology.orchestra.run.smithi178.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:57:43.653 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:57 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:57:43.704 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:57:43.705 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2021-11-20T05:57:43.705 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:43.772 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs-devel 2021-11-20T05:57:44.310 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:59 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:57:44.689 INFO:teuthology.orchestra.run.smithi139.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:57:44.985 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:57:44.985 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-11-20T05:57:44.986 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:57:45.067 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install libcephfs-devel 2021-11-20T05:57:45.605 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:59 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:57:45.612 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:57:45.613 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:45.613 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2021-11-20T05:57:45.613 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:45.614 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:57:45.614 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2021-11-20T05:57:45.614 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:45.614 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:57:45.615 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:57:45.615 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2021-11-20T05:57:45.616 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:45.616 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 24 k 2021-11-20T05:57:45.616 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 94 k 2021-11-20T05:57:45.616 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:57:45.625 INFO:teuthology.orchestra.run.smithi178.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 24 kB 00:00 2021-11-20T05:57:45.625 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:57:45.626 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.9 MB/s | 24 kB 00:00 2021-11-20T05:57:45.626 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:57:45.630 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:57:45.631 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:57:45.632 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:57:45.636 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:57:45.688 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:57:45.887 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:46.127 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:46.690 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:46.691 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:46.691 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:57:46.691 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2021-11-20T05:57:46.692 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:57:46.692 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:46.801 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados2 2021-11-20T05:57:46.895 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:57:46.895 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:46.896 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2021-11-20T05:57:46.896 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:46.896 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:57:46.896 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2021-11-20T05:57:46.896 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:46.897 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:57:46.897 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:57:46.897 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2021-11-20T05:57:46.897 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:46.898 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 24 k 2021-11-20T05:57:46.898 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 94 k 2021-11-20T05:57:46.898 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:57:46.933 INFO:teuthology.orchestra.run.smithi139.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 653 kB/s | 24 kB 00:00 2021-11-20T05:57:46.934 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:57:46.934 INFO:teuthology.orchestra.run.smithi139.stdout:Total 629 kB/s | 24 kB 00:00 2021-11-20T05:57:46.935 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:57:46.938 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:57:46.939 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:57:46.940 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:57:46.944 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:57:47.339 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:02 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:57:47.555 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:57:47.962 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:48.213 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:48.322 INFO:teuthology.orchestra.run.smithi178.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:57:48.501 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:57:48.501 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:48.502 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:57:48.502 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2021-11-20T05:57:48.502 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:57:48.502 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:57:48.610 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:57:48.611 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2021-11-20T05:57:48.612 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:48.624 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librados2 2021-11-20T05:57:48.669 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librbd1 2021-11-20T05:57:49.164 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:03 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:57:49.207 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:04 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:57:50.162 INFO:teuthology.orchestra.run.smithi139.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:57:50.196 INFO:teuthology.orchestra.run.smithi178.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:57:50.450 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:57:50.451 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-11-20T05:57:50.452 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:57:50.485 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:57:50.486 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2021-11-20T05:57:50.486 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:50.529 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librbd1 2021-11-20T05:57:50.544 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rados 2021-11-20T05:57:51.069 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:05 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:57:51.083 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:06 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:57:52.112 INFO:teuthology.orchestra.run.smithi139.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:57:52.401 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:57:52.402 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-11-20T05:57:52.402 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:57:52.470 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rados 2021-11-20T05:57:53.009 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:07 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:57:53.281 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:57:53.571 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:57:53.572 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2021-11-20T05:57:53.573 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:53.647 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rgw 2021-11-20T05:57:54.185 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:09 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:57:54.445 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:57:54.733 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:57:54.734 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-11-20T05:57:54.734 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:57:54.806 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rgw 2021-11-20T05:57:55.250 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:57:55.347 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:09 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:57:55.538 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:57:55.539 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2021-11-20T05:57:55.539 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:55.605 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-cephfs 2021-11-20T05:57:56.145 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:11 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:57:56.336 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:57:56.626 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:57:56.951 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-11-20T05:57:56.951 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:57:56.952 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-cephfs 2021-11-20T05:57:57.220 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:57:57.492 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:11 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:57:57.514 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:57:57.515 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2021-11-20T05:57:57.515 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:57.587 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rbd 2021-11-20T05:57:58.123 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:13 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:57:58.486 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:57:58.777 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:57:58.779 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-11-20T05:57:58.779 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:57:58.855 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rbd 2021-11-20T05:57:59.336 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:57:59.400 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:13 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:57:59.626 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:57:59.627 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2021-11-20T05:57:59.627 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:57:59.714 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-fuse 2021-11-20T05:58:00.317 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:15 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:58:00.419 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-20T05:58:00.707 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:58:00.708 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2021-11-20T05:58:00.708 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:58:00.789 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-fuse 2021-11-20T05:58:01.329 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:15 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:58:01.635 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:58:01.636 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:58:01.636 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2021-11-20T05:58:01.636 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:58:01.636 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:58:01.637 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2021-11-20T05:58:01.637 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:58:01.637 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:58:01.637 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:58:01.638 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2021-11-20T05:58:01.638 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:58:01.638 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 89 k 2021-11-20T05:58:01.638 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 298 k 2021-11-20T05:58:01.639 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:58:01.649 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 7.0 MB/s | 89 kB 00:00 2021-11-20T05:58:01.650 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:58:01.650 INFO:teuthology.orchestra.run.smithi178.stdout:Total 6.2 MB/s | 89 kB 00:00 2021-11-20T05:58:01.651 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:58:01.663 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:58:01.664 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:58:01.716 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:58:01.720 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:58:01.833 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:58:02.061 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:02.334 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:02.668 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:58:02.669 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:58:02.669 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2021-11-20T05:58:02.670 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:58:02.670 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:58:02.670 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2021-11-20T05:58:02.670 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:58:02.671 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:58:02.671 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:58:02.671 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2021-11-20T05:58:02.671 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:58:02.672 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 89 k 2021-11-20T05:58:02.672 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 298 k 2021-11-20T05:58:02.672 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:58:02.674 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2021-11-20T05:58:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:58:02.676 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:58:02.681 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 7.5 MB/s | 89 kB 00:00 2021-11-20T05:58:02.682 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:58:02.683 INFO:teuthology.orchestra.run.smithi139.stdout:Total 6.7 MB/s | 89 kB 00:00 2021-11-20T05:58:02.683 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:58:02.694 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:58:02.695 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:58:02.744 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:58:02.747 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:58:02.848 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:58:02.885 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-mirror 2021-11-20T05:58:03.034 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:03.301 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:03.451 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:18 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:58:03.621 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:03.622 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:58:03.622 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:58:03.622 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2021-11-20T05:58:03.622 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:58:03.623 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:58:03.813 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-mirror 2021-11-20T05:58:04.360 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:18 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:58:04.738 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:58:04.739 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:58:04.740 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2021-11-20T05:58:04.740 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:58:04.740 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:58:04.740 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T05:58:04.741 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:58:04.741 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:58:04.741 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:58:04.741 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2021-11-20T05:58:04.741 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:58:04.742 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 3.7 M 2021-11-20T05:58:04.742 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 18 M 2021-11-20T05:58:04.742 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:58:04.795 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 68 MB/s | 3.7 MB 00:00 2021-11-20T05:58:04.796 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:58:04.796 INFO:teuthology.orchestra.run.smithi178.stdout:Total 65 MB/s | 3.7 MB 00:00 2021-11-20T05:58:04.797 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:58:04.813 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:58:04.814 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:58:04.874 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:58:04.878 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:58:05.339 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:58:05.551 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:05.654 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:58:05.655 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:58:05.656 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2021-11-20T05:58:05.656 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:58:05.656 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:58:05.656 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-20T05:58:05.657 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:58:05.657 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:58:05.657 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:58:05.657 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2021-11-20T05:58:05.658 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:58:05.658 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 3.7 M 2021-11-20T05:58:05.658 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 18 M 2021-11-20T05:58:05.658 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:58:05.710 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 69 MB/s | 3.7 MB 00:00 2021-11-20T05:58:05.711 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:58:05.711 INFO:teuthology.orchestra.run.smithi139.stdout:Total 67 MB/s | 3.7 MB 00:00 2021-11-20T05:58:05.711 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:58:05.726 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:58:05.726 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:58:05.790 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:58:05.793 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:58:05.948 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:06.257 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:58:06.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:06.300 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:58:06.300 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:58:06.300 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2021-11-20T05:58:06.300 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:58:06.301 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:58:06.452 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:06.497 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-nbd 2021-11-20T05:58:06.840 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:07.046 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:22 ago on Sat 20 Nov 2021 05:55:45 AM UTC. 2021-11-20T05:58:07.154 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:07.154 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:58:07.155 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:58:07.155 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2021-11-20T05:58:07.156 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:58:07.156 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:58:07.314 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-nbd 2021-11-20T05:58:07.865 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:21 ago on Sat 20 Nov 2021 05:55:46 AM UTC. 2021-11-20T05:58:08.324 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2021-11-20T05:58:08.325 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:58:08.325 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2021-11-20T05:58:08.326 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:58:08.326 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2021-11-20T05:58:08.326 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2021-11-20T05:58:08.326 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:58:08.327 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2021-11-20T05:58:08.327 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2021-11-20T05:58:08.327 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2021-11-20T05:58:08.327 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:58:08.327 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 173 k 2021-11-20T05:58:08.328 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 649 k 2021-11-20T05:58:08.328 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2021-11-20T05:58:08.339 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 12 MB/s | 173 kB 00:00 2021-11-20T05:58:08.340 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:58:08.340 INFO:teuthology.orchestra.run.smithi178.stdout:Total 11 MB/s | 173 kB 00:00 2021-11-20T05:58:08.341 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2021-11-20T05:58:08.353 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2021-11-20T05:58:08.353 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2021-11-20T05:58:08.404 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2021-11-20T05:58:08.407 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2021-11-20T05:58:08.522 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2021-11-20T05:58:08.759 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:09.034 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:09.182 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2021-11-20T05:58:09.183 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:58:09.183 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2021-11-20T05:58:09.184 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:58:09.184 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2021-11-20T05:58:09.184 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2021-11-20T05:58:09.184 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:58:09.185 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2021-11-20T05:58:09.185 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2021-11-20T05:58:09.185 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2021-11-20T05:58:09.185 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:58:09.186 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 173 k 2021-11-20T05:58:09.186 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 649 k 2021-11-20T05:58:09.186 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2021-11-20T05:58:09.197 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 12 MB/s | 173 kB 00:00 2021-11-20T05:58:09.198 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2021-11-20T05:58:09.199 INFO:teuthology.orchestra.run.smithi139.stdout:Total 11 MB/s | 173 kB 00:00 2021-11-20T05:58:09.199 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2021-11-20T05:58:09.211 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2021-11-20T05:58:09.211 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2021-11-20T05:58:09.262 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2021-11-20T05:58:09.265 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2021-11-20T05:58:09.383 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2021-11-20T05:58:09.386 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:09.386 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:58:09.386 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2021-11-20T05:58:09.387 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2021-11-20T05:58:09.387 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:58:09.387 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2021-11-20T05:58:09.603 DEBUG:teuthology.parallel:result is None 2021-11-20T05:58:09.616 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:09.880 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:10.206 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-20T05:58:10.206 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:58:10.206 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2021-11-20T05:58:10.207 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2021-11-20T05:58:10.207 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T05:58:10.207 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2021-11-20T05:58:10.373 DEBUG:teuthology.parallel:result is None 2021-11-20T05:58:10.374 DEBUG:teuthology.orchestra.run.smithi139:> sudo lsb_release -is 2021-11-20T05:58:10.407 INFO:teuthology.orchestra.run.smithi139.stdout:CentOSStream 2021-11-20T05:58:10.409 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T05:58:10.409 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T05:58:10.410 INFO:teuthology.packaging:ref: None 2021-11-20T05:58:10.410 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T05:58:10.411 INFO:teuthology.packaging:branch: None 2021-11-20T05:58:10.411 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T05:58:10.642 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:58:10.644 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:58:10.863 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-11-20T05:58:10.885 INFO:teuthology.orchestra.run.smithi139.stdout:16.2.4-0.el8 2021-11-20T05:58:10.886 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2021-11-20T05:58:10.887 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2021-11-20T05:58:10.887 DEBUG:teuthology.orchestra.run.smithi178:> sudo lsb_release -is 2021-11-20T05:58:10.929 INFO:teuthology.orchestra.run.smithi178.stdout:CentOSStream 2021-11-20T05:58:10.930 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-20T05:58:10.931 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-20T05:58:10.931 INFO:teuthology.packaging:ref: None 2021-11-20T05:58:10.932 INFO:teuthology.packaging:tag: v16.2.4 2021-11-20T05:58:10.932 INFO:teuthology.packaging:branch: None 2021-11-20T05:58:10.932 INFO:teuthology.packaging:sha1: 9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa 2021-11-20T05:58:11.156 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:58:11.157 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-20T05:58:11.407 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-11-20T05:58:11.432 INFO:teuthology.orchestra.run.smithi178.stdout:16.2.4-0.el8 2021-11-20T05:58:11.433 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2021-11-20T05:58:11.433 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2021-11-20T05:58:11.434 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-11-20T05:58:11.435 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-11-20T05:58:11.435 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-11-20T05:58:11.463 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2021-11-20T05:58:11.463 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-11-20T05:58:11.501 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-11-20T05:58:11.502 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-11-20T05:58:11.502 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/daemon-helper 2021-11-20T05:58:11.529 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-11-20T05:58:11.594 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2021-11-20T05:58:11.595 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2021-11-20T05:58:11.623 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-11-20T05:58:11.691 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-11-20T05:58:11.691 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-11-20T05:58:11.692 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/adjust-ulimits 2021-11-20T05:58:11.717 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-11-20T05:58:11.782 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2021-11-20T05:58:11.782 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2021-11-20T05:58:11.810 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-11-20T05:58:11.877 INFO:teuthology.run_tasks:Running task cephadm... 2021-11-20T05:58:11.955 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa'} 2021-11-20T05:58:11.956 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2021-11-20T05:58:11.957 INFO:tasks.cephadm:Cluster fsid is d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T05:58:11.957 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2021-11-20T05:58:11.958 INFO:tasks.cephadm:No mon roles; fabricating mons 2021-11-20T05:58:11.958 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi139': '172.21.15.139', 'mon.smithi178': '172.21.15.178'} 2021-11-20T05:58:11.958 INFO:tasks.cephadm:Normalizing hostnames... 2021-11-20T05:58:11.959 DEBUG:teuthology.orchestra.run.smithi139:> sudo hostname $(hostname -s) 2021-11-20T05:58:11.985 DEBUG:teuthology.orchestra.run.smithi178:> sudo hostname $(hostname -s) 2021-11-20T05:58:12.012 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2021-11-20T05:58:12.012 DEBUG:teuthology.orchestra.run.smithi139:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2021-11-20T05:58:12.335 INFO:teuthology.orchestra.run.smithi139.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 20 05:58 /home/ubuntu/cephtest/cephadm 2021-11-20T05:58:12.336 DEBUG:teuthology.orchestra.run.smithi178:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2021-11-20T05:58:12.439 INFO:teuthology.orchestra.run.smithi178.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 20 05:58 /home/ubuntu/cephtest/cephadm 2021-11-20T05:58:12.439 DEBUG:teuthology.orchestra.run.smithi139:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-11-20T05:58:12.457 DEBUG:teuthology.orchestra.run.smithi178:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-11-20T05:58:12.488 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /etc/ceph 2021-11-20T05:58:12.522 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph 2021-11-20T05:58:12.551 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 777 /etc/ceph 2021-11-20T05:58:12.587 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /etc/ceph 2021-11-20T05:58:12.617 INFO:tasks.cephadm:Writing seed config... 2021-11-20T05:58:12.619 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2021-11-20T05:58:12.620 INFO:tasks.cephadm: override: [client] debug client = 20 2021-11-20T05:58:12.620 INFO:tasks.cephadm: override: [client] debug ms = 1 2021-11-20T05:58:12.620 INFO:tasks.cephadm: override: [client] rados mon op timeout = 15m 2021-11-20T05:58:12.621 INFO:tasks.cephadm: override: [client] rados osd op timeout = 15m 2021-11-20T05:58:12.621 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2021-11-20T05:58:12.621 INFO:tasks.cephadm: override: [mds] debug mds = 20 2021-11-20T05:58:12.622 INFO:tasks.cephadm: override: [mds] debug ms = 1 2021-11-20T05:58:12.622 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2021-11-20T05:58:12.622 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2021-11-20T05:58:12.623 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2021-11-20T05:58:12.624 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2021-11-20T05:58:12.625 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2021-11-20T05:58:12.625 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 15m 2021-11-20T05:58:12.625 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 15m 2021-11-20T05:58:12.626 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2021-11-20T05:58:12.627 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2021-11-20T05:58:12.627 INFO:tasks.cephadm: override: [mon] debug mon = 20 2021-11-20T05:58:12.628 INFO:tasks.cephadm: override: [mon] debug ms = 1 2021-11-20T05:58:12.628 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2021-11-20T05:58:12.628 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2021-11-20T05:58:12.629 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2021-11-20T05:58:12.629 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2021-11-20T05:58:12.630 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2021-11-20T05:58:12.630 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2021-11-20T05:58:12.630 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2021-11-20T05:58:12.631 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2021-11-20T05:58:12.631 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2021-11-20T05:58:12.631 INFO:tasks.cephadm: override: [osd] debug ms = 1 2021-11-20T05:58:12.632 INFO:tasks.cephadm: override: [osd] debug osd = 20 2021-11-20T05:58:12.632 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2021-11-20T05:58:12.632 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2021-11-20T05:58:12.632 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2021-11-20T05:58:12.633 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2021-11-20T05:58:12.633 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2021-11-20T05:58:12.633 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2021-11-20T05:58:12.634 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2021-11-20T05:58:12.635 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-11-20T05:58:12.635 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2021-11-20T05:58:12.651 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d819f2cc-49c6-11ec-8c2c-001a4aab830c mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 15m rados osd op timeout = 15m [mds] debug mds = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 15m rados osd op timeout = 15m 2021-11-20T05:58:12.651 DEBUG:teuthology.orchestra.run.smithi139:mon.smithi139> sudo journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi139.service 2021-11-20T05:58:12.693 INFO:tasks.cephadm:Bootstrapping... 2021-11-20T05:58:12.694 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.139 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2021-11-20T05:58:12.717 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:-- Logs begin at Sat 2021-11-20 05:47:58 UTC. -- 2021-11-20T05:58:12.942 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-11-20T05:58:13.821 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: 3.4.1-dev 2021-11-20T05:58:13.886 INFO:teuthology.orchestra.run.smithi139.stderr:Verifying podman|docker is present... 2021-11-20T05:58:13.886 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-11-20T05:58:13.929 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: 3.4.1-dev 2021-11-20T05:58:14.032 INFO:teuthology.orchestra.run.smithi139.stderr:Verifying lvm2 is present... 2021-11-20T05:58:14.033 INFO:teuthology.orchestra.run.smithi139.stderr:Verifying time synchronization is in place... 2021-11-20T05:58:14.033 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-enabled chrony.service 2021-11-20T05:58:14.039 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-11-20T05:58:14.040 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-active chrony.service 2021-11-20T05:58:14.045 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: inactive 2021-11-20T05:58:14.046 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-enabled chronyd.service 2021-11-20T05:58:14.052 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: enabled 2021-11-20T05:58:14.053 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-active chronyd.service 2021-11-20T05:58:14.057 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: active 2021-11-20T05:58:14.058 INFO:teuthology.orchestra.run.smithi139.stderr:Unit chronyd.service is enabled and running 2021-11-20T05:58:14.058 INFO:teuthology.orchestra.run.smithi139.stderr:Repeating the final host check... 2021-11-20T05:58:14.059 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-11-20T05:58:14.096 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: 3.4.1-dev 2021-11-20T05:58:14.196 INFO:teuthology.orchestra.run.smithi139.stderr:podman|docker (/bin/podman) is present 2021-11-20T05:58:14.196 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl is present 2021-11-20T05:58:14.196 INFO:teuthology.orchestra.run.smithi139.stderr:lvcreate is present 2021-11-20T05:58:14.197 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-enabled chrony.service 2021-11-20T05:58:14.202 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-11-20T05:58:14.202 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-active chrony.service 2021-11-20T05:58:14.207 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: inactive 2021-11-20T05:58:14.207 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-enabled chronyd.service 2021-11-20T05:58:14.213 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: enabled 2021-11-20T05:58:14.213 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-active chronyd.service 2021-11-20T05:58:14.218 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: active 2021-11-20T05:58:14.219 INFO:teuthology.orchestra.run.smithi139.stderr:Unit chronyd.service is enabled and running 2021-11-20T05:58:14.219 INFO:teuthology.orchestra.run.smithi139.stderr:Host looks OK 2021-11-20T05:58:14.219 INFO:teuthology.orchestra.run.smithi139.stderr:Cluster fsid: d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T05:58:14.220 INFO:teuthology.orchestra.run.smithi139.stderr:Acquiring lock 140214527003336 on /run/cephadm/d819f2cc-49c6-11ec-8c2c-001a4aab830c.lock 2021-11-20T05:58:14.220 INFO:teuthology.orchestra.run.smithi139.stderr:Lock 140214527003336 acquired on /run/cephadm/d819f2cc-49c6-11ec-8c2c-001a4aab830c.lock 2021-11-20T05:58:14.221 INFO:teuthology.orchestra.run.smithi139.stderr:Verifying IP 172.21.15.139 port 3300 ... 2021-11-20T05:58:14.221 INFO:teuthology.orchestra.run.smithi139.stderr:Verifying IP 172.21.15.139 port 6789 ... 2021-11-20T05:58:14.221 INFO:teuthology.orchestra.run.smithi139.stderr:Base mon IP is 172.21.15.139, final addrv is [v2:172.21.15.139:3300,v1:172.21.15.139:6789] 2021-11-20T05:58:14.222 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /sbin/ip route ls 2021-11-20T05:58:14.222 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2021-11-20T05:58:14.222 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.139 metric 100 2021-11-20T05:58:14.223 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /sbin/ip -6 route ls 2021-11-20T05:58:14.224 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2021-11-20T05:58:14.225 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2021-11-20T05:58:14.225 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium 2021-11-20T05:58:14.225 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /sbin/ip -6 addr ls 2021-11-20T05:58:14.227 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2021-11-20T05:58:14.227 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: inet6 ::1/128 scope host 2021-11-20T05:58:14.228 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-11-20T05:58:14.228 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2021-11-20T05:58:14.228 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:7f59/64 scope link noprefixroute 2021-11-20T05:58:14.228 INFO:teuthology.orchestra.run.smithi139.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-11-20T05:58:14.229 INFO:teuthology.orchestra.run.smithi139.stderr:Mon IP 172.21.15.139 is in CIDR network 172.21.0.0/20 2021-11-20T05:58:14.229 INFO:teuthology.orchestra.run.smithi139.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2021-11-20T05:58:14.230 INFO:teuthology.orchestra.run.smithi139.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2021-11-20T05:58:14.230 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2021-11-20T05:58:14.299 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2021-11-20T05:58:14.667 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Getting image source signatures 2021-11-20T05:58:14.683 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2021-11-20T05:58:14.683 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2021-11-20T05:58:14.779 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2021-11-20T05:58:14.837 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2021-11-20T05:58:26.835 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2021-11-20T05:58:26.902 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Writing manifest to image destination 2021-11-20T05:58:26.903 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: Storing signatures 2021-11-20T05:58:27.208 INFO:teuthology.orchestra.run.smithi139.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2021-11-20T05:58:27.298 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2021-11-20T05:58:27.802 INFO:teuthology.orchestra.run.smithi139.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2021-11-20T05:58:28.142 INFO:teuthology.orchestra.run.smithi139.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2021-11-20T05:58:28.142 INFO:teuthology.orchestra.run.smithi139.stderr:Extracting ceph user uid/gid from container image... 2021-11-20T05:58:28.142 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2021-11-20T05:58:28.519 INFO:teuthology.orchestra.run.smithi139.stderr:stat: 167 167 2021-11-20T05:58:28.885 INFO:teuthology.orchestra.run.smithi139.stderr:Creating initial keys... 2021-11-20T05:58:28.886 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2021-11-20T05:58:29.235 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-authtool: AQCFjphhFMBgDhAAVJevbl2sBps5Rd4SiXRpYg== 2021-11-20T05:58:29.628 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2021-11-20T05:58:29.976 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-authtool: AQCFjphhqqoOOhAAPZUTloYKURfE10QNQ9C0dQ== 2021-11-20T05:58:30.273 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2021-11-20T05:58:30.659 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-authtool: AQCGjphhDTtcJxAAR5iHnCVS3lwaN0GY8d+e4Q== 2021-11-20T05:58:31.021 INFO:teuthology.orchestra.run.smithi139.stderr:Creating initial monmap... 2021-11-20T05:58:31.021 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpnnz8bbxh:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c --addv smithi139 [v2:172.21.15.139:3300,v1:172.21.15.139:6789] /tmp/monmap 2021-11-20T05:58:31.427 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2021-11-20T05:58:31.427 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T05:58:31.427 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-11-20T05:58:31.863 INFO:teuthology.orchestra.run.smithi139.stderr:monmaptool for smithi139 [v2:172.21.15.139:3300,v1:172.21.15.139:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2021-11-20T05:58:31.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/monmaptool: set fsid to d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T05:58:31.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-11-20T05:58:31.863 INFO:teuthology.orchestra.run.smithi139.stderr: 2021-11-20T05:58:31.864 INFO:teuthology.orchestra.run.smithi139.stderr:Creating mon... 2021-11-20T05:58:31.865 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139:/var/lib/ceph/mon/ceph-smithi139:z -v /tmp/ceph-tmpvxpkkox2:/tmp/keyring:z -v /tmp/ceph-tmpnnz8bbxh:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi139 --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2021-11-20T05:58:32.226 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.234+0000 7f0a5a7f1700 0 set uid:gid to 167:167 (ceph:ceph) 2021-11-20T05:58:32.255 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.234+0000 7f0a5a7f1700 1 imported monmap: 2021-11-20T05:58:32.255 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: epoch 0 2021-11-20T05:58:32.255 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T05:58:32.256 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: last_changed 2021-11-20T05:58:31.415306+0000 2021-11-20T05:58:32.256 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: created 2021-11-20T05:58:31.415306+0000 2021-11-20T05:58:32.256 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2021-11-20T05:58:32.256 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: election_strategy: 1 2021-11-20T05:58:32.257 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon.smithi139 2021-11-20T05:58:32.257 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.257 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.234+0000 7f0a5a7f1700 0 /usr/bin/ceph-mon: set fsid to d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T05:58:32.257 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: RocksDB version: 6.8.1 2021-11-20T05:58:32.257 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.258 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-11-20T05:58:32.258 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Compile date May 13 2021 2021-11-20T05:58:32.258 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: DB SUMMARY 2021-11-20T05:58:32.258 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.259 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi139/store.db dir, Total Num: 0, files: 2021-11-20T05:58:32.259 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.259 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi139/store.db: 2021-11-20T05:58:32.259 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.259 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.error_if_exists: 0 2021-11-20T05:58:32.260 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.create_if_missing: 1 2021-11-20T05:58:32.260 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.paranoid_checks: 1 2021-11-20T05:58:32.260 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.env: 0x55c53740b1c0 2021-11-20T05:58:32.260 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.fs: Posix File System 2021-11-20T05:58:32.261 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.info_log: 0x55c53947cec0 2021-11-20T05:58:32.261 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.max_file_opening_threads: 16 2021-11-20T05:58:32.261 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.statistics: (nil) 2021-11-20T05:58:32.261 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.use_fsync: 0 2021-11-20T05:58:32.261 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.max_log_file_size: 0 2021-11-20T05:58:32.262 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-11-20T05:58:32.262 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-11-20T05:58:32.262 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.keep_log_file_num: 1000 2021-11-20T05:58:32.262 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.recycle_log_file_num: 0 2021-11-20T05:58:32.263 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.allow_fallocate: 1 2021-11-20T05:58:32.263 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.allow_mmap_reads: 0 2021-11-20T05:58:32.263 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.allow_mmap_writes: 0 2021-11-20T05:58:32.263 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.use_direct_reads: 0 2021-11-20T05:58:32.263 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-11-20T05:58:32.264 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.create_missing_column_families: 0 2021-11-20T05:58:32.264 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.db_log_dir: 2021-11-20T05:58:32.264 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi139/store.db 2021-11-20T05:58:32.264 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-11-20T05:58:32.265 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.max_subcompactions: 1 2021-11-20T05:58:32.265 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.max_background_flushes: -1 2021-11-20T05:58:32.265 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-11-20T05:58:32.265 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-11-20T05:58:32.265 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-11-20T05:58:32.266 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-11-20T05:58:32.266 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-11-20T05:58:32.266 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.advise_random_on_open: 1 2021-11-20T05:58:32.266 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.db_write_buffer_size: 0 2021-11-20T05:58:32.267 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.write_buffer_manager: 0x55c539483530 2021-11-20T05:58:32.267 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-11-20T05:58:32.267 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-11-20T05:58:32.267 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-11-20T05:58:32.267 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-11-20T05:58:32.268 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.rate_limiter: (nil) 2021-11-20T05:58:32.268 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-11-20T05:58:32.268 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.wal_recovery_mode: 2 2021-11-20T05:58:32.268 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.enable_thread_tracking: 0 2021-11-20T05:58:32.269 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.enable_pipelined_write: 0 2021-11-20T05:58:32.269 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.unordered_write: 0 2021-11-20T05:58:32.269 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-11-20T05:58:32.269 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-11-20T05:58:32.269 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-11-20T05:58:32.270 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-11-20T05:58:32.270 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.row_cache: None 2021-11-20T05:58:32.270 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.wal_filter: None 2021-11-20T05:58:32.270 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-11-20T05:58:32.271 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.allow_ingest_behind: 0 2021-11-20T05:58:32.271 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.preserve_deletes: 0 2021-11-20T05:58:32.271 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.two_write_queues: 0 2021-11-20T05:58:32.271 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.manual_wal_flush: 0 2021-11-20T05:58:32.271 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.atomic_flush: 0 2021-11-20T05:58:32.272 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-11-20T05:58:32.272 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-11-20T05:58:32.272 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-11-20T05:58:32.272 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.log_readahead_size: 0 2021-11-20T05:58:32.273 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-11-20T05:58:32.273 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.max_background_jobs: 2 2021-11-20T05:58:32.273 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.max_background_compactions: -1 2021-11-20T05:58:32.273 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-11-20T05:58:32.273 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-11-20T05:58:32.274 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-11-20T05:58:32.274 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.max_total_wal_size: 0 2021-11-20T05:58:32.274 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-11-20T05:58:32.274 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-11-20T05:58:32.275 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-11-20T05:58:32.275 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-11-20T05:58:32.275 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.max_open_files: -1 2021-11-20T05:58:32.275 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.bytes_per_sync: 0 2021-11-20T05:58:32.275 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-11-20T05:58:32.276 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-11-20T05:58:32.276 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Options.compaction_readahead_size: 0 2021-11-20T05:58:32.276 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Compression algorithms supported: 2021-11-20T05:58:32.276 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-11-20T05:58:32.277 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: kZSTD supported: 0 2021-11-20T05:58:32.277 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: kXpressCompression supported: 0 2021-11-20T05:58:32.277 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: kLZ4HCCompression supported: 1 2021-11-20T05:58:32.277 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: kLZ4Compression supported: 1 2021-11-20T05:58:32.277 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: kBZip2Compression supported: 0 2021-11-20T05:58:32.278 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: kZlibCompression supported: 1 2021-11-20T05:58:32.278 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: kSnappyCompression supported: 1 2021-11-20T05:58:32.278 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-11-20T05:58:32.278 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.235+0000 7f0a5a7f1700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2021-11-20T05:58:32.279 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.279 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi139/store.db/MANIFEST-000001 2021-11-20T05:58:32.279 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.279 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-11-20T05:58:32.279 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.280 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-11-20T05:58:32.280 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.merge_operator: 2021-11-20T05:58:32.280 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compaction_filter: None 2021-11-20T05:58:32.280 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compaction_filter_factory: None 2021-11-20T05:58:32.281 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-11-20T05:58:32.281 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-11-20T05:58:32.281 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c539394120) 2021-11-20T05:58:32.281 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2021-11-20T05:58:32.281 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2021-11-20T05:58:32.282 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2021-11-20T05:58:32.282 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2021-11-20T05:58:32.282 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: index_type: 0 2021-11-20T05:58:32.282 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2021-11-20T05:58:32.282 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: index_shortening: 1 2021-11-20T05:58:32.283 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2021-11-20T05:58:32.283 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2021-11-20T05:58:32.283 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: checksum: 1 2021-11-20T05:58:32.283 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2021-11-20T05:58:32.284 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: block_cache: 0x55c5393cad10 2021-11-20T05:58:32.284 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2021-11-20T05:58:32.284 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: block_cache_options: 2021-11-20T05:58:32.284 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: capacity : 536870912 2021-11-20T05:58:32.284 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2021-11-20T05:58:32.285 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2021-11-20T05:58:32.285 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2021-11-20T05:58:32.285 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2021-11-20T05:58:32.285 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2021-11-20T05:58:32.285 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: block_size: 4096 2021-11-20T05:58:32.286 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2021-11-20T05:58:32.286 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2021-11-20T05:58:32.286 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2021-11-20T05:58:32.286 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2021-11-20T05:58:32.287 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: partition_filters: 0 2021-11-20T05:58:32.287 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2021-11-20T05:58:32.287 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2021-11-20T05:58:32.287 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2021-11-20T05:58:32.287 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: verify_compression: 0 2021-11-20T05:58:32.288 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2021-11-20T05:58:32.288 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: format_version: 2 2021-11-20T05:58:32.288 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2021-11-20T05:58:32.288 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: block_align: 0 2021-11-20T05:58:32.289 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.289 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.write_buffer_size: 33554432 2021-11-20T05:58:32.289 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.max_write_buffer_number: 2 2021-11-20T05:58:32.289 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compression: NoCompression 2021-11-20T05:58:32.289 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.bottommost_compression: Disabled 2021-11-20T05:58:32.290 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.prefix_extractor: nullptr 2021-11-20T05:58:32.290 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-11-20T05:58:32.290 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.num_levels: 7 2021-11-20T05:58:32.290 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-11-20T05:58:32.291 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-11-20T05:58:32.291 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-11-20T05:58:32.291 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-11-20T05:58:32.291 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-11-20T05:58:32.291 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-11-20T05:58:32.292 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-11-20T05:58:32.292 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-11-20T05:58:32.292 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-11-20T05:58:32.292 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-11-20T05:58:32.292 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compression_opts.level: 32767 2021-11-20T05:58:32.293 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compression_opts.strategy: 0 2021-11-20T05:58:32.293 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-11-20T05:58:32.293 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-11-20T05:58:32.293 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compression_opts.enabled: false 2021-11-20T05:58:32.294 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-11-20T05:58:32.294 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-11-20T05:58:32.294 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-11-20T05:58:32.294 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.target_file_size_base: 67108864 2021-11-20T05:58:32.294 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-11-20T05:58:32.295 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-11-20T05:58:32.295 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-11-20T05:58:32.295 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-11-20T05:58:32.295 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-11-20T05:58:32.295 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-11-20T05:58:32.296 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-11-20T05:58:32.296 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-11-20T05:58:32.296 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-11-20T05:58:32.296 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-11-20T05:58:32.297 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-11-20T05:58:32.297 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-11-20T05:58:32.297 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-11-20T05:58:32.297 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.arena_block_size: 4194304 2021-11-20T05:58:32.297 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-11-20T05:58:32.298 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-11-20T05:58:32.298 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-11-20T05:58:32.298 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.disable_auto_compactions: 0 2021-11-20T05:58:32.298 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-11-20T05:58:32.299 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-11-20T05:58:32.299 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-11-20T05:58:32.299 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-11-20T05:58:32.299 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-11-20T05:58:32.300 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-11-20T05:58:32.300 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-11-20T05:58:32.300 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-11-20T05:58:32.300 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-11-20T05:58:32.300 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-11-20T05:58:32.301 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.table_properties_collectors: 2021-11-20T05:58:32.301 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.inplace_update_support: 0 2021-11-20T05:58:32.301 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-11-20T05:58:32.301 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-11-20T05:58:32.301 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-11-20T05:58:32.302 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-11-20T05:58:32.302 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.bloom_locality: 0 2021-11-20T05:58:32.302 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.max_successive_merges: 0 2021-11-20T05:58:32.302 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-11-20T05:58:32.303 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.paranoid_file_checks: 0 2021-11-20T05:58:32.303 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.force_consistency_checks: 0 2021-11-20T05:58:32.303 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.report_bg_io_stats: 0 2021-11-20T05:58:32.303 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.ttl: 2592000 2021-11-20T05:58:32.303 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.238+0000 7f0a5a7f1700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-11-20T05:58:32.304 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.240+0000 7f0a5a7f1700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi139/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-11-20T05:58:32.304 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.304 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.240+0000 7f0a5a7f1700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-11-20T05:58:32.304 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.305 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.241+0000 7f0a5a7f1700 4 rocksdb: DB pointer 0x55c53948f800 2021-11-20T05:58:32.305 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.241+0000 7f0a4368a700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-11-20T05:58:32.305 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.241+0000 7f0a4368a700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-11-20T05:58:32.305 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2021-11-20T05:58:32.305 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T05:58:32.306 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-11-20T05:58:32.306 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-11-20T05:58:32.306 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T05:58:32.306 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-11-20T05:58:32.307 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-11-20T05:58:32.307 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T05:58:32.307 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.307 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T05:58:32.307 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:58:32.308 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:58:32.308 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T05:58:32.308 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T05:58:32.308 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.308 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T05:58:32.309 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:58:32.309 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:58:32.309 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T05:58:32.309 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T05:58:32.309 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T05:58:32.310 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T05:58:32.310 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T05:58:32.310 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-11-20T05:58:32.310 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:58:32.311 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:58:32.311 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T05:58:32.311 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.311 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-11-20T05:58:32.311 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.312 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T05:58:32.312 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:58:32.312 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:58:32.312 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T05:58:32.313 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T05:58:32.313 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.313 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-20T05:58:32.313 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:58:32.314 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:58:32.314 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T05:58:32.314 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T05:58:32.314 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T05:58:32.315 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T05:58:32.315 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T05:58:32.315 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-11-20T05:58:32.315 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:58:32.315 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:58:32.316 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T05:58:32.316 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.316 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-11-20T05:58:32.316 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: 2021-11-20T05:58:32.316 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.243+0000 7f0a5a7f1700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2021-11-20T05:58:32.317 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.243+0000 7f0a5a7f1700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2021-11-20T05:58:32.317 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph-mon: debug 2021-11-20T05:58:32.243+0000 7f0a5a7f1700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi139 for mon.smithi139 2021-11-20T05:58:32.605 INFO:teuthology.orchestra.run.smithi139.stderr:create mon.smithi139 on 2021-11-20T05:58:32.606 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T05:58:32.611 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl disable ceph.target 2021-11-20T05:58:32.618 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2021-11-20T05:58:32.700 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl enable ceph.target 2021-11-20T05:58:32.707 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2021-11-20T05:58:32.783 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl start ceph.target 2021-11-20T05:58:32.790 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl enable ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c.target 2021-11-20T05:58:32.795 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c.target → /etc/systemd/system/ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c.target. 2021-11-20T05:58:32.796 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c.target → /etc/systemd/system/ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c.target. 2021-11-20T05:58:32.870 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl start ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c.target 2021-11-20T05:58:32.876 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl daemon-reload 2021-11-20T05:58:32.959 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl stop ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi139 2021-11-20T05:58:32.987 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl reset-failed ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi139 2021-11-20T05:58:32.992 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Failed to reset failed state of unit ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi139.service: Unit ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi139.service not loaded. 2021-11-20T05:58:32.992 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl enable ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi139 2021-11-20T05:58:32.999 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Created symlink /etc/systemd/system/ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c.target.wants/ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi139.service → /etc/systemd/system/ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@.service. 2021-11-20T05:58:33.077 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl start ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi139 2021-11-20T05:58:33.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:33 smithi139 systemd[1]: Starting Ceph mon.smithi139 for d819f2cc-49c6-11ec-8c2c-001a4aab830c... 2021-11-20T05:58:33.508 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T05:58:33.516 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: disabled 2021-11-20T05:58:33.516 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T05:58:33.522 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: inactive 2021-11-20T05:58:33.523 INFO:teuthology.orchestra.run.smithi139.stderr:firewalld.service is not enabled 2021-11-20T05:58:33.524 INFO:teuthology.orchestra.run.smithi139.stderr:Not possible to enable service . firewalld.service is not available 2021-11-20T05:58:33.524 INFO:teuthology.orchestra.run.smithi139.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-11-20T05:58:33.525 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for mon to start... 2021-11-20T05:58:33.525 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for mon... 2021-11-20T05:58:33.525 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139:/var/lib/ceph/mon/ceph-smithi139:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2021-11-20T05:58:33.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:33 smithi139 podman[32766]: 2021-11-20T05:58:33.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:33 smithi139 bash[32688]: 3bf94e1a3b4a6d72e09949dee67f61ece9ce7f514ed2645c53169831729eacb0 2021-11-20T05:58:33.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:33 smithi139 systemd[1]: Started Ceph mon.smithi139 for d819f2cc-49c6-11ec-8c2c-001a4aab830c. 2021-11-20T05:58:33.959 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:33 smithi139 conmon[32806]: cluster 2021-11-20T05:58:33.540281+0000 mon.smithi139 (mon.0) 0 : [INF] mkfs d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T05:58:33.959 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:33 smithi139 conmon[32806]: cluster 2021-11-20T05:58:33 2021-11-20T05:58:33.959 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:33 smithi139 conmon[32806]: .514037+0000 mon.smithi139 (mon.0) 1 : cluster [INF] mon.smithi139 is new leader, mons smithi139 in quorum (ranks 0) 2021-11-20T05:58:33.961 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.971+0000 7f766db82700 1 Processor -- start 2021-11-20T05:58:33.962 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.971+0000 7f766db82700 1 -- start start 2021-11-20T05:58:33.962 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.971+0000 7f766db82700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766806fbd0 0x7f7668100c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:33.962 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.971+0000 7f766db82700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76680fc680 con 0x7f766806fbd0 2021-11-20T05:58:33.962 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.971+0000 7f76677fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766806fbd0 0x7f7668100c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:33.963 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.971+0000 7f76677fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766806fbd0 0x7f7668100c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56404/0 (socket says 172.21.15.139:56404) 2021-11-20T05:58:33.963 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.971+0000 7f76677fe700 1 -- 172.21.15.139:0/1695377147 learned_addr learned my addr 172.21.15.139:0/1695377147 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:33.963 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.973+0000 7f76677fe700 1 -- 172.21.15.139:0/1695377147 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7668101180 con 0x7f766806fbd0 2021-11-20T05:58:33.963 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.973+0000 7f76677fe700 1 --2- 172.21.15.139:0/1695377147 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766806fbd0 0x7f7668100c90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7658009960 tx=0x7f76580092d0).ready entity=mon.0 client_cookie=c7f7a07584b6b24 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:33.964 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.973+0000 7f76667fc700 1 -- 172.21.15.139:0/1695377147 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f765800b9c0 con 0x7f766806fbd0 2021-11-20T05:58:33.964 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.973+0000 7f76667fc700 1 -- 172.21.15.139:0/1695377147 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765800bb20 con 0x7f766806fbd0 2021-11-20T05:58:33.964 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.973+0000 7f76667fc700 1 -- 172.21.15.139:0/1695377147 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f765800bdf0 con 0x7f766806fbd0 2021-11-20T05:58:33.964 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.973+0000 7f766db82700 1 -- 172.21.15.139:0/1695377147 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766806fbd0 msgr2=0x7f7668100c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:33.964 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.973+0000 7f766db82700 1 --2- 172.21.15.139:0/1695377147 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766806fbd0 0x7f7668100c90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7658009960 tx=0x7f76580092d0).stop 2021-11-20T05:58:33.965 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.973+0000 7f766db82700 1 -- 172.21.15.139:0/1695377147 shutdown_connections 2021-11-20T05:58:33.965 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.973+0000 7f766db82700 1 --2- 172.21.15.139:0/1695377147 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766806fbd0 0x7f7668100c90 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:33.965 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.973+0000 7f766db82700 1 -- 172.21.15.139:0/1695377147 >> 172.21.15.139:0/1695377147 conn(0x7f76680f6310 msgr2=0x7f76680f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:33.965 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.973+0000 7f766db82700 1 -- 172.21.15.139:0/1695377147 shutdown_connections 2021-11-20T05:58:33.966 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.973+0000 7f766db82700 1 -- 172.21.15.139:0/1695377147 wait complete. 2021-11-20T05:58:33.966 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.973+0000 7f766db82700 1 Processor -- start 2021-11-20T05:58:33.966 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.973+0000 7f766db82700 1 -- start start 2021-11-20T05:58:33.966 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.975+0000 7f766db82700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766806fbd0 0x7f766810c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:33.966 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.975+0000 7f766db82700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766810c890 con 0x7f766806fbd0 2021-11-20T05:58:33.967 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.975+0000 7f76677fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766806fbd0 0x7f766810c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:33.967 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.975+0000 7f76677fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766806fbd0 0x7f766810c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56406/0 (socket says 172.21.15.139:56406) 2021-11-20T05:58:33.967 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.975+0000 7f76677fe700 1 -- 172.21.15.139:0/4010209418 learned_addr learned my addr 172.21.15.139:0/4010209418 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:33.967 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.975+0000 7f76677fe700 1 -- 172.21.15.139:0/4010209418 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7658009570 con 0x7f766806fbd0 2021-11-20T05:58:33.968 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.975+0000 7f76677fe700 1 --2- 172.21.15.139:0/4010209418 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766806fbd0 0x7f766810c3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f7658009f20 tx=0x7f7658017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:33.969 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.975+0000 7f764ffff700 1 -- 172.21.15.139:0/4010209418 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f765801c420 con 0x7f766806fbd0 2021-11-20T05:58:33.969 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.975+0000 7f764ffff700 1 -- 172.21.15.139:0/4010209418 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765801c580 con 0x7f766806fbd0 2021-11-20T05:58:33.969 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.975+0000 7f766db82700 1 -- 172.21.15.139:0/4010209418 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766810ca90 con 0x7f766806fbd0 2021-11-20T05:58:33.969 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.975+0000 7f764ffff700 1 -- 172.21.15.139:0/4010209418 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f765801c850 con 0x7f766806fbd0 2021-11-20T05:58:33.969 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.975+0000 7f766db82700 1 -- 172.21.15.139:0/4010209418 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766810cea0 con 0x7f766806fbd0 2021-11-20T05:58:33.970 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.976+0000 7f764ffff700 1 -- 172.21.15.139:0/4010209418 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f765802c070 con 0x7f766806fbd0 2021-11-20T05:58:33.970 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.976+0000 7f764ffff700 1 -- 172.21.15.139:0/4010209418 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7658025620 con 0x7f766806fbd0 2021-11-20T05:58:33.970 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.976+0000 7f766db82700 1 -- 172.21.15.139:0/4010209418 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7668045ad0 con 0x7f766806fbd0 2021-11-20T05:58:33.970 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:33.979+0000 7f764ffff700 1 -- 172.21.15.139:0/4010209418 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f765802a060 con 0x7f766806fbd0 2021-11-20T05:58:34.034 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.045+0000 7f766db82700 1 -- 172.21.15.139:0/4010209418 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f7668105f90 con 0x7f766806fbd0 2021-11-20T05:58:34.034 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.045+0000 7f764ffff700 1 -- 172.21.15.139:0/4010209418 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f7658020070 con 0x7f766806fbd0 2021-11-20T05:58:34.034 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: cluster: 2021-11-20T05:58:34.034 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: id: d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T05:58:34.035 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: health: HEALTH_OK 2021-11-20T05:58:34.035 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.035 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: services: 2021-11-20T05:58:34.035 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi139 (age 0.510112s) 2021-11-20T05:58:34.036 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mgr: no daemons active 2021-11-20T05:58:34.036 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2021-11-20T05:58:34.036 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.036 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: data: 2021-11-20T05:58:34.036 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2021-11-20T05:58:34.037 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2021-11-20T05:58:34.037 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2021-11-20T05:58:34.037 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: pgs: 2021-11-20T05:58:34.038 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.038 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.046+0000 7f766db82700 1 -- 172.21.15.139:0/4010209418 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766806fbd0 msgr2=0x7f766810c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:34.038 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.046+0000 7f766db82700 1 --2- 172.21.15.139:0/4010209418 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766806fbd0 0x7f766810c3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f7658009f20 tx=0x7f7658017000).stop 2021-11-20T05:58:34.038 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.046+0000 7f766db82700 1 -- 172.21.15.139:0/4010209418 shutdown_connections 2021-11-20T05:58:34.039 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.046+0000 7f766db82700 1 --2- 172.21.15.139:0/4010209418 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f766806fbd0 0x7f766810c3a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:34.039 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.046+0000 7f766db82700 1 -- 172.21.15.139:0/4010209418 >> 172.21.15.139:0/4010209418 conn(0x7f76680f6310 msgr2=0x7f76680f8020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:34.039 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.046+0000 7f766db82700 1 -- 172.21.15.139:0/4010209418 shutdown_connections 2021-11-20T05:58:34.039 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.046+0000 7f766db82700 1 -- 172.21.15.139:0/4010209418 wait complete. 2021-11-20T05:58:34.363 INFO:teuthology.orchestra.run.smithi139.stderr:mon is available 2021-11-20T05:58:34.363 INFO:teuthology.orchestra.run.smithi139.stderr:Assimilating anything we can from ceph.conf... 2021-11-20T05:58:34.363 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z -v /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139:/var/lib/ceph/mon/ceph-smithi139:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi139/config 2021-11-20T05:58:34.555 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:34 smithi139 conmon[32806]: cluster 2021-11-20T05:58:33.536057+0000 mon.smithi139 (mon.0) 2 : cluster [INF] mon.smithi139 is new leader, mons smithi139 in quorum (ranks 0) 2021-11-20T05:58:34.555 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:34 smithi139 conmon[32806]: cluster 2021-11-20T05:58:33.536290+0000 mon.smithi139 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi139=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} 2021-11-20T05:58:34.556 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:34 smithi139 conmon[32806]: cluster 2021-11-20T05:58:33.545106+0000 mon.smithi139 (mon.0) 4 : cluster [DBG] fsmap 2021-11-20T05:58:34.556 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:34 smithi139 conmon[32806]: cluster 2021-11-20T05:58:33.547274+0000 mon.smithi139 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-11-20T05:58:34.556 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:34 smithi139 conmon[32806]: cluster 2021-11-20T05:58:33.547748+0000 mon.smithi139 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2021-11-20T05:58:34.556 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:34 smithi139 conmon[32806]: audit 2021-11-20T05:58:34.046133+0000 mon.smithi139 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.139:0/4010209418' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2021-11-20T05:58:34.783 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.792+0000 7fd6a3ce9700 1 Processor -- start 2021-11-20T05:58:34.784 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.792+0000 7fd6a3ce9700 1 -- start start 2021-11-20T05:58:34.784 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.793+0000 7fd6a3ce9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd69c0fbbc0 0x7fd69c0fbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:34.784 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.793+0000 7fd6a3ce9700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c0fc4d0 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.785 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.793+0000 7fd6a1a85700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd69c0fbbc0 0x7fd69c0fbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:34.785 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.793+0000 7fd6a1a85700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd69c0fbbc0 0x7fd69c0fbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56408/0 (socket says 172.21.15.139:56408) 2021-11-20T05:58:34.785 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.793+0000 7fd6a1a85700 1 -- 172.21.15.139:0/2681946475 learned_addr learned my addr 172.21.15.139:0/2681946475 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:34.785 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.793+0000 7fd6a1a85700 1 -- 172.21.15.139:0/2681946475 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c0fc610 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.786 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.793+0000 7fd6a1a85700 1 --2- 172.21.15.139:0/2681946475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd69c0fbbc0 0x7fd69c0fbfe0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fd69800b5f0 tx=0x7fd698006ee0).ready entity=mon.0 client_cookie=9a8fcea2c4518e34 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:34.786 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.793+0000 7fd6a0a83700 1 -- 172.21.15.139:0/2681946475 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd69800d600 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.786 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.793+0000 7fd6a0a83700 1 -- 172.21.15.139:0/2681946475 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69800d760 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.786 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.794+0000 7fd6a0a83700 1 -- 172.21.15.139:0/2681946475 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd69800da30 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.786 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.794+0000 7fd6a3ce9700 1 -- 172.21.15.139:0/2681946475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd69c0fbbc0 msgr2=0x7fd69c0fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:34.787 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.794+0000 7fd6a3ce9700 1 --2- 172.21.15.139:0/2681946475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd69c0fbbc0 0x7fd69c0fbfe0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fd69800b5f0 tx=0x7fd698006ee0).stop 2021-11-20T05:58:34.787 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.794+0000 7fd6a3ce9700 1 -- 172.21.15.139:0/2681946475 shutdown_connections 2021-11-20T05:58:34.787 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.794+0000 7fd6a3ce9700 1 --2- 172.21.15.139:0/2681946475 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd69c0fbbc0 0x7fd69c0fbfe0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:34.788 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.794+0000 7fd6a3ce9700 1 -- 172.21.15.139:0/2681946475 >> 172.21.15.139:0/2681946475 conn(0x7fd69c0f73e0 msgr2=0x7fd69c0f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:34.788 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.794+0000 7fd6a3ce9700 1 -- 172.21.15.139:0/2681946475 shutdown_connections 2021-11-20T05:58:34.788 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.794+0000 7fd6a3ce9700 1 -- 172.21.15.139:0/2681946475 wait complete. 2021-11-20T05:58:34.788 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.795+0000 7fd6a3ce9700 1 Processor -- start 2021-11-20T05:58:34.788 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.795+0000 7fd6a3ce9700 1 -- start start 2021-11-20T05:58:34.789 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.795+0000 7fd6a3ce9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd69c0fbbc0 0x7fd69c10a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:34.789 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.795+0000 7fd6a3ce9700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c10a6b0 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.789 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.795+0000 7fd6a1a85700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd69c0fbbc0 0x7fd69c10a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:34.789 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.795+0000 7fd6a1a85700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd69c0fbbc0 0x7fd69c10a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56410/0 (socket says 172.21.15.139:56410) 2021-11-20T05:58:34.790 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.795+0000 7fd6a1a85700 1 -- 172.21.15.139:0/2855696567 learned_addr learned my addr 172.21.15.139:0/2855696567 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:34.790 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.796+0000 7fd6a1a85700 1 -- 172.21.15.139:0/2855696567 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69800b200 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.790 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.796+0000 7fd6a1a85700 1 --2- 172.21.15.139:0/2855696567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd69c0fbbc0 0x7fd69c10a1c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fd698006c70 tx=0x7fd698016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:34.790 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.796+0000 7fd68a7fc700 1 -- 172.21.15.139:0/2855696567 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd698011b40 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.791 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.796+0000 7fd68a7fc700 1 -- 172.21.15.139:0/2855696567 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd698011ca0 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.791 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.796+0000 7fd6a3ce9700 1 -- 172.21.15.139:0/2855696567 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69c10a8b0 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.791 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.796+0000 7fd68a7fc700 1 -- 172.21.15.139:0/2855696567 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd69801d430 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.791 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.796+0000 7fd6a3ce9700 1 -- 172.21.15.139:0/2855696567 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69c10acc0 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.791 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.796+0000 7fd68a7fc700 1 -- 172.21.15.139:0/2855696567 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd69800f070 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.792 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.796+0000 7fd68a7fc700 1 -- 172.21.15.139:0/2855696567 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd698027460 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.792 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.797+0000 7fd6a3ce9700 1 -- 172.21.15.139:0/2855696567 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd69c045ad0 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.793 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.799+0000 7fd68a7fc700 1 -- 172.21.15.139:0/2855696567 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd698016070 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.849 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.862+0000 7fd6a3ce9700 1 -- 172.21.15.139:0/2855696567 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fd69c0fcf00 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.855 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.866+0000 7fd68a7fc700 1 -- 172.21.15.139:0/2855696567 <== mon.0 v2:172.21.15.139:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd69801d5d0 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.855 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.866+0000 7fd68a7fc700 1 -- 172.21.15.139:0/2855696567 <== mon.0 v2:172.21.15.139:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+466 (secure 0 0 0) 0x7fd6980276f0 con 0x7fd69c0fbbc0 2021-11-20T05:58:34.855 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.856 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: [global] 2021-11-20T05:58:34.856 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: fsid = d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T05:58:34.856 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.139:3300,v1:172.21.15.139:6789] 2021-11-20T05:58:34.856 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2021-11-20T05:58:34.856 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2021-11-20T05:58:34.857 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2021-11-20T05:58:34.857 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2021-11-20T05:58:34.857 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.857 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: [mgr] 2021-11-20T05:58:34.858 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2021-11-20T05:58:34.858 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.858 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: [osd] 2021-11-20T05:58:34.858 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2021-11-20T05:58:34.858 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2021-11-20T05:58:34.859 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2021-11-20T05:58:34.859 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2021-11-20T05:58:34.859 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2021-11-20T05:58:34.859 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2021-11-20T05:58:34.860 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.867+0000 7fd6a3ce9700 1 -- 172.21.15.139:0/2855696567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd69c0fbbc0 msgr2=0x7fd69c10a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:34.860 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.867+0000 7fd6a3ce9700 1 --2- 172.21.15.139:0/2855696567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd69c0fbbc0 0x7fd69c10a1c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fd698006c70 tx=0x7fd698016000).stop 2021-11-20T05:58:34.860 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.867+0000 7fd6a3ce9700 1 -- 172.21.15.139:0/2855696567 shutdown_connections 2021-11-20T05:58:34.860 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.867+0000 7fd6a3ce9700 1 --2- 172.21.15.139:0/2855696567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd69c0fbbc0 0x7fd69c10a1c0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:34.861 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.867+0000 7fd6a3ce9700 1 -- 172.21.15.139:0/2855696567 >> 172.21.15.139:0/2855696567 conn(0x7fd69c0f73e0 msgr2=0x7fd69c0fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:34.861 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.867+0000 7fd6a3ce9700 1 -- 172.21.15.139:0/2855696567 shutdown_connections 2021-11-20T05:58:34.861 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:34.867+0000 7fd6a3ce9700 1 -- 172.21.15.139:0/2855696567 wait complete. 2021-11-20T05:58:35.198 INFO:teuthology.orchestra.run.smithi139.stderr:Generating new minimal ceph.conf... 2021-11-20T05:58:35.198 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z -v /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139:/var/lib/ceph/mon/ceph-smithi139:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi139/config 2021-11-20T05:58:35.638 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.646+0000 7f53f085c700 1 Processor -- start 2021-11-20T05:58:35.638 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.646+0000 7f53f085c700 1 -- start start 2021-11-20T05:58:35.638 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.646+0000 7f53f085c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53ec06fce0 0x7f53ec0fe840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:35.639 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.646+0000 7f53f085c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec0fed30 con 0x7f53ec06fce0 2021-11-20T05:58:35.639 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.647+0000 7f53ea59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53ec06fce0 0x7f53ec0fe840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:35.639 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.647+0000 7f53ea59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53ec06fce0 0x7f53ec0fe840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56412/0 (socket says 172.21.15.139:56412) 2021-11-20T05:58:35.639 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.647+0000 7f53ea59c700 1 -- 172.21.15.139:0/1679520868 learned_addr learned my addr 172.21.15.139:0/1679520868 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:35.640 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.647+0000 7f53ea59c700 1 -- 172.21.15.139:0/1679520868 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec0fee70 con 0x7f53ec06fce0 2021-11-20T05:58:35.640 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.647+0000 7f53ea59c700 1 --2- 172.21.15.139:0/1679520868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53ec06fce0 0x7f53ec0fe840 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f53dc00b5f0 tx=0x7f53dc006ee0).ready entity=mon.0 client_cookie=979b86b968e8452d server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:35.640 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.647+0000 7f53e9d9b700 1 -- 172.21.15.139:0/1679520868 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53dc00d600 con 0x7f53ec06fce0 2021-11-20T05:58:35.640 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.647+0000 7f53e9d9b700 1 -- 172.21.15.139:0/1679520868 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f53dc00dbd0 con 0x7f53ec06fce0 2021-11-20T05:58:35.640 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.647+0000 7f53e9d9b700 1 -- 172.21.15.139:0/1679520868 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53dc011c00 con 0x7f53ec06fce0 2021-11-20T05:58:35.641 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.648+0000 7f53f085c700 1 -- 172.21.15.139:0/1679520868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53ec06fce0 msgr2=0x7f53ec0fe840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:35.641 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.648+0000 7f53f085c700 1 --2- 172.21.15.139:0/1679520868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53ec06fce0 0x7f53ec0fe840 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f53dc00b5f0 tx=0x7f53dc006ee0).stop 2021-11-20T05:58:35.641 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.648+0000 7f53f085c700 1 -- 172.21.15.139:0/1679520868 shutdown_connections 2021-11-20T05:58:35.641 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.648+0000 7f53f085c700 1 --2- 172.21.15.139:0/1679520868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53ec06fce0 0x7f53ec0fe840 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:35.642 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.648+0000 7f53f085c700 1 -- 172.21.15.139:0/1679520868 >> 172.21.15.139:0/1679520868 conn(0x7f53ec0f62e0 msgr2=0x7f53ec0f8700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:35.642 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.648+0000 7f53f085c700 1 -- 172.21.15.139:0/1679520868 shutdown_connections 2021-11-20T05:58:35.642 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.648+0000 7f53f085c700 1 -- 172.21.15.139:0/1679520868 wait complete. 2021-11-20T05:58:35.642 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.649+0000 7f53f085c700 1 Processor -- start 2021-11-20T05:58:35.642 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.649+0000 7f53f085c700 1 -- start start 2021-11-20T05:58:35.643 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.649+0000 7f53f085c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53ec10a9f0 0x7f53ec10ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:35.643 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.649+0000 7f53f085c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ec10b300 con 0x7f53ec10a9f0 2021-11-20T05:58:35.643 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.649+0000 7f53ea59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53ec10a9f0 0x7f53ec10ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:35.643 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.649+0000 7f53ea59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53ec10a9f0 0x7f53ec10ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56414/0 (socket says 172.21.15.139:56414) 2021-11-20T05:58:35.644 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.649+0000 7f53ea59c700 1 -- 172.21.15.139:0/42836639 learned_addr learned my addr 172.21.15.139:0/42836639 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:35.644 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.649+0000 7f53ea59c700 1 -- 172.21.15.139:0/42836639 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53dc00b200 con 0x7f53ec10a9f0 2021-11-20T05:58:35.644 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.649+0000 7f53ea59c700 1 --2- 172.21.15.139:0/42836639 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53ec10a9f0 0x7f53ec10ae10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f53dc00b190 tx=0x7f53dc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:35.644 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.649+0000 7f53e37fe700 1 -- 172.21.15.139:0/42836639 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53dc01b410 con 0x7f53ec10a9f0 2021-11-20T05:58:35.644 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.650+0000 7f53e37fe700 1 -- 172.21.15.139:0/42836639 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f53dc01b9e0 con 0x7f53ec10a9f0 2021-11-20T05:58:35.645 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.650+0000 7f53f085c700 1 -- 172.21.15.139:0/42836639 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53ec10b500 con 0x7f53ec10a9f0 2021-11-20T05:58:35.645 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.650+0000 7f53e37fe700 1 -- 172.21.15.139:0/42836639 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53dc01a880 con 0x7f53ec10a9f0 2021-11-20T05:58:35.645 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.650+0000 7f53f085c700 1 -- 172.21.15.139:0/42836639 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53ec10bfd0 con 0x7f53ec10a9f0 2021-11-20T05:58:35.645 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.650+0000 7f53e37fe700 1 -- 172.21.15.139:0/42836639 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f53dc00f070 con 0x7f53ec10a9f0 2021-11-20T05:58:35.645 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.650+0000 7f53e37fe700 1 -- 172.21.15.139:0/42836639 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f53dc025c90 con 0x7f53ec10a9f0 2021-11-20T05:58:35.646 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.651+0000 7f53f085c700 1 -- 172.21.15.139:0/42836639 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d0004fa0 con 0x7f53ec10a9f0 2021-11-20T05:58:35.646 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.653+0000 7f53e37fe700 1 -- 172.21.15.139:0/42836639 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f53dc016070 con 0x7f53ec10a9f0 2021-11-20T05:58:35.703 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.717+0000 7f53f085c700 1 -- 172.21.15.139:0/42836639 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f53d0005670 con 0x7f53ec10a9f0 2021-11-20T05:58:35.704 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.717+0000 7f53e37fe700 1 -- 172.21.15.139:0/42836639 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f53dc01bb50 con 0x7f53ec10a9f0 2021-11-20T05:58:35.705 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.718+0000 7f53f085c700 1 -- 172.21.15.139:0/42836639 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53ec10a9f0 msgr2=0x7f53ec10ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:35.705 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.718+0000 7f53f085c700 1 --2- 172.21.15.139:0/42836639 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53ec10a9f0 0x7f53ec10ae10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f53dc00b190 tx=0x7f53dc006ee0).stop 2021-11-20T05:58:35.706 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.718+0000 7f53f085c700 1 -- 172.21.15.139:0/42836639 shutdown_connections 2021-11-20T05:58:35.706 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.718+0000 7f53f085c700 1 --2- 172.21.15.139:0/42836639 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53ec10a9f0 0x7f53ec10ae10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:35.706 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.718+0000 7f53f085c700 1 -- 172.21.15.139:0/42836639 >> 172.21.15.139:0/42836639 conn(0x7f53ec0f62e0 msgr2=0x7f53ec0f7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:35.706 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.718+0000 7f53f085c700 1 -- 172.21.15.139:0/42836639 shutdown_connections 2021-11-20T05:58:35.707 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:35.718+0000 7f53f085c700 1 -- 172.21.15.139:0/42836639 wait complete. 2021-11-20T05:58:35.868 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:35 smithi139 conmon[32806]: audit 2021-11-20T05:58:34.863832+0000 mon.smithi139 (mon. 2021-11-20T05:58:35.869 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:35 smithi139 conmon[32806]: 0) 8 : audit [INF] from='client.? 172.21.15.139:0/2855696567' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-11-20T05:58:35.869 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:35 smithi139 conmon[32806]: audit 2021-11-20T05:58:34.866872+0000 mon.smithi139 (mon.0) 9 : audit [INF] from='client.? 172.21.15.139:0/2855696567' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-11-20T05:58:36.034 INFO:teuthology.orchestra.run.smithi139.stderr:Restarting the monitor... 2021-11-20T05:58:36.034 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl restart ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi139 2021-11-20T05:58:36.283 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:36 smithi139 systemd[1]: Stopping Ceph mon.smithi139 for d819f2cc-49c6-11ec-8c2c-001a4aab830c... 2021-11-20T05:58:36.283 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:36 smithi139 conmon[32806]: debug 2021-11-20T05:58:36.129+0000 7f90d2dcc700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi139 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-11-20T05:58:36.284 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:36 smithi139 conmon[32806]: debug 2021-11-20T05:58:36.129+0000 7f90d2dcc700 -1 mon.smithi139@0(leader) e1 *** Got Signal Terminated *** 2021-11-20T05:58:36.284 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:36 smithi139 conmon[32806]: 2021-11-20T05:58:36.696 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:36 smithi139 podman[33294]: ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c-mon.smithi139 2021-11-20T05:58:37.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:36 smithi139 systemd[1]: ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi139.service: Succeeded. 2021-11-20T05:58:37.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:36 smithi139 systemd[1]: Stopped Ceph mon.smithi139 for d819f2cc-49c6-11ec-8c2c-001a4aab830c. 2021-11-20T05:58:37.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:36 smithi139 systemd[1]: Starting Ceph mon.smithi139 for d819f2cc-49c6-11ec-8c2c-001a4aab830c... 2021-11-20T05:58:37.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:36 smithi139 podman[33461]: 2021-11-20T05:58:37.116 INFO:teuthology.orchestra.run.smithi139.stderr:Setting mon public_network to 172.21.0.0/20 2021-11-20T05:58:37.116 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2021-11-20T05:58:37.349 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 bash[33385]: e8a76ef704b8fb71cfc325ed34b8e8e1f68a21b7e0380d2c46389433c6d25a31 2021-11-20T05:58:37.349 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.105+0000 7f2a35832700 0 set uid:gid to 167:167 (ceph:ceph) 2021-11-20T05:58:37.349 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.105+0000 7f2a35832700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2021-11-20T05:58:37.349 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.105+0000 7f2a35832700 0 pidfile_write: ignore empty --pid-file 2021-11-20T05:58:37.350 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.109+0000 7f2a35832700 0 load: jerasure load: lrc load: isa 2021-11-20T05:58:37.350 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.109+0000 7f2a35832700 4 rocksdb: RocksDB version: 6.8.1 2021-11-20T05:58:37.350 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.351 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.109+0000 7f2a35832700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-11-20T05:58:37.351 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.109+0000 7f2a35832700 4 rocksdb: Compile date May 13 2021 2021-11-20T05:58:37.351 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.109+0000 7f2a35832700 4 rocksdb: DB SUMMARY 2021-11-20T05:58:37.351 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.352 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.109+0000 7f2a35832700 4 rocksdb: CURRENT file: CURRENT 2021-11-20T05:58:37.352 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.352 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.109+0000 7f2a35832700 4 rocksdb: IDENTITY file: IDENTITY 2021-11-20T05:58:37.352 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.353 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2021-11-20T05:58:37.353 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.353 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi139/store.db dir, Total Num: 1, files: 000004.sst 2021-11-20T05:58:37.353 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.354 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi139/store.db: 000006.log size: 92488 ; 2021-11-20T05:58:37.354 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.354 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.error_if_exists: 0 2021-11-20T05:58:37.354 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.create_if_missing: 0 2021-11-20T05:58:37.354 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.paranoid_checks: 1 2021-11-20T05:58:37.355 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.env: 0x55f8c46c51c0 2021-11-20T05:58:37.355 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.fs: Posix File System 2021-11-20T05:58:37.355 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.info_log: 0x55f8c67ad960 2021-11-20T05:58:37.355 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_file_opening_threads: 16 2021-11-20T05:58:37.356 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.statistics: (nil) 2021-11-20T05:58:37.356 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.use_fsync: 0 2021-11-20T05:58:37.356 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_log_file_size: 0 2021-11-20T05:58:37.356 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-11-20T05:58:37.356 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-11-20T05:58:37.357 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.keep_log_file_num: 1000 2021-11-20T05:58:37.357 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.recycle_log_file_num: 0 2021-11-20T05:58:37.357 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.allow_fallocate: 1 2021-11-20T05:58:37.357 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.allow_mmap_reads: 0 2021-11-20T05:58:37.358 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.allow_mmap_writes: 0 2021-11-20T05:58:37.358 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.use_direct_reads: 0 2021-11-20T05:58:37.358 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-11-20T05:58:37.358 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.create_missing_column_families: 0 2021-11-20T05:58:37.359 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.db_log_dir: 2021-11-20T05:58:37.359 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi139/store.db 2021-11-20T05:58:37.359 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-11-20T05:58:37.359 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_subcompactions: 1 2021-11-20T05:58:37.359 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_background_flushes: -1 2021-11-20T05:58:37.360 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-11-20T05:58:37.360 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-11-20T05:58:37.360 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-11-20T05:58:37.360 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-11-20T05:58:37.361 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-11-20T05:58:37.361 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.advise_random_on_open: 1 2021-11-20T05:58:37.361 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.db_write_buffer_size: 0 2021-11-20T05:58:37.362 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.write_buffer_manager: 0x55f8c67b3fb0 2021-11-20T05:58:37.362 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-11-20T05:58:37.362 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-11-20T05:58:37.362 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-11-20T05:58:37.363 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-11-20T05:58:37.363 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.rate_limiter: (nil) 2021-11-20T05:58:37.363 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-11-20T05:58:37.364 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.wal_recovery_mode: 2 2021-11-20T05:58:37.364 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.enable_thread_tracking: 0 2021-11-20T05:58:37.364 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.enable_pipelined_write: 0 2021-11-20T05:58:37.364 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.unordered_write: 0 2021-11-20T05:58:37.364 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-11-20T05:58:37.365 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-11-20T05:58:37.365 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-11-20T05:58:37.365 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-11-20T05:58:37.365 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.row_cache: None 2021-11-20T05:58:37.366 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.wal_filter: None 2021-11-20T05:58:37.366 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-11-20T05:58:37.366 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.allow_ingest_behind: 0 2021-11-20T05:58:37.366 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.preserve_deletes: 0 2021-11-20T05:58:37.367 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.two_write_queues: 0 2021-11-20T05:58:37.367 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.manual_wal_flush: 0 2021-11-20T05:58:37.367 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.atomic_flush: 0 2021-11-20T05:58:37.367 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-11-20T05:58:37.367 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-11-20T05:58:37.368 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-11-20T05:58:37.368 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.log_readahead_size: 0 2021-11-20T05:58:37.368 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-11-20T05:58:37.368 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_background_jobs: 2 2021-11-20T05:58:37.369 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_background_compactions: -1 2021-11-20T05:58:37.369 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-11-20T05:58:37.369 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-11-20T05:58:37.369 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-11-20T05:58:37.369 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_total_wal_size: 0 2021-11-20T05:58:37.370 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-11-20T05:58:37.370 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-11-20T05:58:37.370 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-11-20T05:58:37.370 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-11-20T05:58:37.371 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_open_files: -1 2021-11-20T05:58:37.371 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.bytes_per_sync: 0 2021-11-20T05:58:37.371 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-11-20T05:58:37.371 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-11-20T05:58:37.371 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compaction_readahead_size: 0 2021-11-20T05:58:37.372 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Compression algorithms supported: 2021-11-20T05:58:37.372 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-11-20T05:58:37.372 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: kZSTD supported: 0 2021-11-20T05:58:37.372 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: kXpressCompression supported: 0 2021-11-20T05:58:37.373 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: kLZ4HCCompression supported: 1 2021-11-20T05:58:37.373 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: kLZ4Compression supported: 1 2021-11-20T05:58:37.373 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: kBZip2Compression supported: 0 2021-11-20T05:58:37.373 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: kZlibCompression supported: 1 2021-11-20T05:58:37.373 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: kSnappyCompression supported: 1 2021-11-20T05:58:37.374 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-11-20T05:58:37.374 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi139/store.db/MANIFEST-000005 2021-11-20T05:58:37.374 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.374 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-11-20T05:58:37.375 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.375 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-11-20T05:58:37.375 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.merge_operator: 2021-11-20T05:58:37.375 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compaction_filter: None 2021-11-20T05:58:37.375 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compaction_filter_factory: None 2021-11-20T05:58:37.377 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-11-20T05:58:37.377 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-11-20T05:58:37.378 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.378 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f8c66c4128) 2021-11-20T05:58:37.378 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: cache_index_and_filter_blocks: 1 2021-11-20T05:58:37.378 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: cache_index_and_filter_blocks_with_high_priority: 0 2021-11-20T05:58:37.378 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-11-20T05:58:37.379 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: pin_top_level_index_and_filter: 1 2021-11-20T05:58:37.379 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: index_type: 0 2021-11-20T05:58:37.379 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: data_block_index_type: 0 2021-11-20T05:58:37.379 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: index_shortening: 1 2021-11-20T05:58:37.379 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: data_block_hash_table_util_ratio: 0.750000 2021-11-20T05:58:37.380 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: hash_index_allow_collision: 1 2021-11-20T05:58:37.380 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: checksum: 1 2021-11-20T05:58:37.380 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: no_block_cache: 0 2021-11-20T05:58:37.380 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: block_cache: 0x55f8c66faf10 2021-11-20T05:58:37.380 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: block_cache_name: BinnedLRUCache 2021-11-20T05:58:37.381 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: block_cache_options: 2021-11-20T05:58:37.381 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: capacity : 536870912 2021-11-20T05:58:37.381 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: num_shard_bits : 4 2021-11-20T05:58:37.381 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: strict_capacity_limit : 0 2021-11-20T05:58:37.382 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: high_pri_pool_ratio: 0.000 2021-11-20T05:58:37.382 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: block_cache_compressed: (nil) 2021-11-20T05:58:37.382 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: persistent_cache: (nil) 2021-11-20T05:58:37.382 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: block_size: 4096 2021-11-20T05:58:37.382 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: block_size_deviation: 10 2021-11-20T05:58:37.383 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: block_restart_interval: 16 2021-11-20T05:58:37.383 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: index_block_restart_interval: 1 2021-11-20T05:58:37.383 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: metadata_block_size: 4096 2021-11-20T05:58:37.383 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: partition_filters: 0 2021-11-20T05:58:37.384 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: use_delta_encoding: 1 2021-11-20T05:58:37.384 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: filter_policy: rocksdb.BuiltinBloomFilter 2021-11-20T05:58:37.384 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: whole_key_filtering: 1 2021-11-20T05:58:37.384 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: verify_compression: 0 2021-11-20T05:58:37.384 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: read_amp_bytes_per_bit: 0 2021-11-20T05:58:37.385 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: format_version: 2 2021-11-20T05:58:37.385 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: enable_index_compression: 1 2021-11-20T05:58:37.385 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: block_align: 0 2021-11-20T05:58:37.385 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.385 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.write_buffer_size: 33554432 2021-11-20T05:58:37.386 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_write_buffer_number: 2 2021-11-20T05:58:37.386 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compression: NoCompression 2021-11-20T05:58:37.386 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.bottommost_compression: Disabled 2021-11-20T05:58:37.386 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.prefix_extractor: nullptr 2021-11-20T05:58:37.387 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-11-20T05:58:37.387 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.num_levels: 7 2021-11-20T05:58:37.387 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-11-20T05:58:37.387 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-11-20T05:58:37.387 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-11-20T05:58:37.388 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-11-20T05:58:37.388 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-11-20T05:58:37.388 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-11-20T05:58:37.388 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-11-20T05:58:37.389 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-11-20T05:58:37.389 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-11-20T05:58:37.389 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-11-20T05:58:37.389 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compression_opts.level: 32767 2021-11-20T05:58:37.390 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compression_opts.strategy: 0 2021-11-20T05:58:37.390 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-11-20T05:58:37.390 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-11-20T05:58:37.390 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compression_opts.enabled: false 2021-11-20T05:58:37.390 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-11-20T05:58:37.391 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-11-20T05:58:37.391 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-11-20T05:58:37.391 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.target_file_size_base: 67108864 2021-11-20T05:58:37.391 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-11-20T05:58:37.392 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-11-20T05:58:37.392 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-11-20T05:58:37.392 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-11-20T05:58:37.392 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-11-20T05:58:37.392 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.393 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-11-20T05:58:37.393 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-11-20T05:58:37.393 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-11-20T05:58:37.393 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-11-20T05:58:37.394 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-11-20T05:58:37.394 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-11-20T05:58:37.394 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-11-20T05:58:37.394 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-11-20T05:58:37.395 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.arena_block_size: 4194304 2021-11-20T05:58:37.395 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-11-20T05:58:37.395 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-11-20T05:58:37.395 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-11-20T05:58:37.395 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.disable_auto_compactions: 0 2021-11-20T05:58:37.396 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-11-20T05:58:37.396 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-11-20T05:58:37.396 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-11-20T05:58:37.396 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-11-20T05:58:37.397 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-11-20T05:58:37.397 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-11-20T05:58:37.397 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-11-20T05:58:37.397 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-11-20T05:58:37.397 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-11-20T05:58:37.398 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-11-20T05:58:37.398 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.table_properties_collectors: 2021-11-20T05:58:37.398 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.inplace_update_support: 0 2021-11-20T05:58:37.399 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-11-20T05:58:37.399 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-11-20T05:58:37.399 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-11-20T05:58:37.399 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-11-20T05:58:37.399 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.bloom_locality: 0 2021-11-20T05:58:37.400 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.max_successive_merges: 0 2021-11-20T05:58:37.400 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-11-20T05:58:37.400 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.paranoid_file_checks: 0 2021-11-20T05:58:37.400 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.force_consistency_checks: 0 2021-11-20T05:58:37.401 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.report_bg_io_stats: 0 2021-11-20T05:58:37.401 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.ttl: 2592000 2021-11-20T05:58:37.401 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-11-20T05:58:37.401 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi139/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-11-20T05:58:37.401 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.402 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2021-11-20T05:58:37.402 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.402 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637387917111450, "job": 1, "event": "recovery_started", "log_files": [6]} 2021-11-20T05:58:37.402 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.402 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.110+0000 7f2a35832700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2021-11-20T05:58:37.403 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.111+0000 7f2a35832700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-11-20T05:58:37.403 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.111+0000 7f2a35832700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637387917112300, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88288, "table_properties": {"data_size": 86415, "index_size": 350, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12269, "raw_average_key_size": 49, "raw_value_size": 79629, "raw_average_value_size": 322, "num_data_blocks": 11, "num_entries": 247, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637387917, "oldest_key_time": 3, "file_creation_time": 0}} 2021-11-20T05:58:37.403 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.111+0000 7f2a35832700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2021-11-20T05:58:37.403 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.404 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.112+0000 7f2a35832700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637387917112675, "job": 1, "event": "recovery_finished"} 2021-11-20T05:58:37.404 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.112+0000 7f2a35832700 4 rocksdb: DB pointer 0x55f8c67bf800 2021-11-20T05:58:37.404 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.112+0000 7f2a1c64a700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-11-20T05:58:37.404 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.405 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.112+0000 7f2a1c64a700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-11-20T05:58:37.405 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: ** DB Stats ** 2021-11-20T05:58:37.405 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T05:58:37.405 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-11-20T05:58:37.405 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-11-20T05:58:37.406 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T05:58:37.406 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-11-20T05:58:37.406 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-11-20T05:58:37.406 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T05:58:37.407 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.407 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: ** Compaction Stats [default] ** 2021-11-20T05:58:37.407 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:58:37.407 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:58:37.407 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: L0 2/0 87.91 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.2 0.00 0.00 1 0.001 0 0 2021-11-20T05:58:37.408 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Sum 2/0 87.91 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.2 0.00 0.00 1 0.001 0 0 2021-11-20T05:58:37.408 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.2 0.00 0.00 1 0.001 0 0 2021-11-20T05:58:37.408 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.408 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: ** Compaction Stats [default] ** 2021-11-20T05:58:37.409 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:58:37.409 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:58:37.409 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 147.2 0.00 0.00 1 0.001 0 0 2021-11-20T05:58:37.409 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T05:58:37.409 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T05:58:37.410 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T05:58:37.410 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T05:58:37.410 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T05:58:37.410 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T05:58:37.411 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Cumulative compaction: 0.00 GB write, 32.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:58:37.411 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Interval compaction: 0.00 GB write, 32.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:58:37.411 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T05:58:37.411 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.411 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T05:58:37.412 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.412 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: ** Compaction Stats [default] ** 2021-11-20T05:58:37.412 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:58:37.412 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:58:37.413 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: L0 2/0 87.91 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.2 0.00 0.00 1 0.001 0 0 2021-11-20T05:58:37.413 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Sum 2/0 87.91 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.2 0.00 0.00 1 0.001 0 0 2021-11-20T05:58:37.413 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T05:58:37.413 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.413 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: ** Compaction Stats [default] ** 2021-11-20T05:58:37.414 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T05:58:37.414 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T05:58:37.414 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 147.2 0.00 0.00 1 0.001 0 0 2021-11-20T05:58:37.414 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T05:58:37.415 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T05:58:37.415 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T05:58:37.415 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T05:58:37.415 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T05:58:37.415 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T05:58:37.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Cumulative compaction: 0.00 GB write, 31.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:58:37.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T05:58:37.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T05:58:37.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.417 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T05:58:37.417 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.417 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.113+0000 7f2a35832700 0 starting mon.smithi139 rank 0 at public addrs [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] at bind addrs [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi139 fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T05:58:37.417 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.114+0000 7f2a35832700 1 mon.smithi139@-1(???) e1 preinit fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T05:58:37.417 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.114+0000 7f2a35832700 0 mon.smithi139@-1(???).mds e1 new map 2021-11-20T05:58:37.418 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.114+0000 7f2a35832700 0 mon.smithi139@-1(???).mds e1 print_map 2021-11-20T05:58:37.418 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: e1 2021-11-20T05:58:37.418 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T05:58:37.418 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T05:58:37.419 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: legacy client fscid: -1 2021-11-20T05:58:37.419 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.419 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: No filesystems configured 2021-11-20T05:58:37.419 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.419 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.114+0000 7f2a35832700 0 mon.smithi139@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2021-11-20T05:58:37.420 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.114+0000 7f2a35832700 0 mon.smithi139@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T05:58:37.420 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.114+0000 7f2a35832700 0 mon.smithi139@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T05:58:37.420 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.114+0000 7f2a35832700 0 mon.smithi139@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T05:58:37.420 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: audit 2021-11-20T05:58:37.420 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:34.863832+0000 2021-11-20T05:58:37.421 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: mon.smithi139 (mon.0) 8 : audit [INF] from='client.? 172.21.15.139:0/2855696567' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-11-20T05:58:37.421 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:37.421 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: audit 2021-11-20T05:58:37.421 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 2021-11-20T05:58:34.866872+0000 mon.smithi139 (mon 2021-11-20T05:58:37.422 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: .0) 9 : audit [INF] from='client.? 172.21.15.139:0/2855696567' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-11-20T05:58:37.422 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.114+0000 7f2a35832700 1 mon.smithi139@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2021-11-20T05:58:37.422 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.115+0000 7f2a35832700 20 expand_channel_meta expand map: {default=false} 2021-11-20T05:58:37.422 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.115+0000 7f2a35832700 20 expand_channel_meta from 'false' to 'false' 2021-11-20T05:58:37.422 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.115+0000 7f2a35832700 20 expand_channel_meta expanded map: {default=false} 2021-11-20T05:58:37.423 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.115+0000 7f2a35832700 20 expand_channel_meta expand map: {default=info} 2021-11-20T05:58:37.423 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.115+0000 7f2a35832700 20 expand_channel_meta from 'info' to 'info' 2021-11-20T05:58:37.423 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.115+0000 7f2a35832700 20 expand_channel_meta expanded map: {default=info} 2021-11-20T05:58:37.423 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.115+0000 7f2a35832700 20 expand_channel_meta expand map: {default=daemon} 2021-11-20T05:58:37.424 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.115+0000 7f2a35832700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-11-20T05:58:37.424 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.115+0000 7f2a35832700 20 expand_channel_meta expanded map: {default=daemon} 2021-11-20T05:58:37.424 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.115+0000 7f2a35832700 20 expand_channel_meta expand map: {default=debug} 2021-11-20T05:58:37.424 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.115+0000 7f2a35832700 20 expand_channel_meta from 'debug' to 'debug' 2021-11-20T05:58:37.424 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: debug 2021-11-20T05:58:37.115+0000 7f2a35832700 20 expand_channel_meta expanded map: {default=debug} 2021-11-20T05:58:37.425 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 systemd[1]: Started Ceph mon.smithi139 for d819f2cc-49c6-11ec-8c2c-001a4aab830c. 2021-11-20T05:58:37.425 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: cluster 2021-11-20T05:58:37.131463+0000 mon.smithi139 (mon. 2021-11-20T05:58:37.425 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: 0) 1 : cluster [INF] mon.smithi139 is new leader, mons smithi139 in quorum (ranks 0) 2021-11-20T05:58:37.425 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: cluster 2021-11-20T05:58:37.131554+0000 mon.smithi139 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi139=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} 2021-11-20T05:58:37.426 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: cluster 2021-11-20T05:58:37.131607+0000 mon.smithi139 (mon.0) 3 : cluster [DBG] fsmap 2021-11-20T05:58:37.426 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: cluster 2021-11-20T05:58:37.131655+0000 mon.smithi139 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-11-20T05:58:37.426 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:37 smithi139 conmon[33501]: cluster 2021-11-20T05:58:37.132162+0000 mon.smithi139 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2021-11-20T05:58:37.621 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.630+0000 7f645372a700 1 Processor -- start 2021-11-20T05:58:37.621 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.630+0000 7f645372a700 1 -- start start 2021-11-20T05:58:37.622 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.630+0000 7f645372a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f644c0fa020 0x7f644c0fa440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:37.622 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.630+0000 7f645372a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c0fa930 con 0x7f644c0fa020 2021-11-20T05:58:37.622 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.631+0000 7f64514c6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f644c0fa020 0x7f644c0fa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:37.622 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.631+0000 7f64514c6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f644c0fa020 0x7f644c0fa440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56416/0 (socket says 172.21.15.139:56416) 2021-11-20T05:58:37.623 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.631+0000 7f64514c6700 1 -- 172.21.15.139:0/3727554342 learned_addr learned my addr 172.21.15.139:0/3727554342 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:37.623 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.631+0000 7f64514c6700 1 -- 172.21.15.139:0/3727554342 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c0faa70 con 0x7f644c0fa020 2021-11-20T05:58:37.623 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.631+0000 7f64514c6700 1 --2- 172.21.15.139:0/3727554342 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f644c0fa020 0x7f644c0fa440 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f644800b5f0 tx=0x7f6448006ee0).ready entity=mon.0 client_cookie=7b32f9943274e90b server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:37.623 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.631+0000 7f643bfff700 1 -- 172.21.15.139:0/3727554342 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f644800d600 con 0x7f644c0fa020 2021-11-20T05:58:37.624 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.631+0000 7f643bfff700 1 -- 172.21.15.139:0/3727554342 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f644800dbd0 con 0x7f644c0fa020 2021-11-20T05:58:37.624 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.631+0000 7f643bfff700 1 -- 172.21.15.139:0/3727554342 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6448011c00 con 0x7f644c0fa020 2021-11-20T05:58:37.624 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.632+0000 7f645372a700 1 -- 172.21.15.139:0/3727554342 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f644c0fa020 msgr2=0x7f644c0fa440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:37.624 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.632+0000 7f645372a700 1 --2- 172.21.15.139:0/3727554342 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f644c0fa020 0x7f644c0fa440 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f644800b5f0 tx=0x7f6448006ee0).stop 2021-11-20T05:58:37.624 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.632+0000 7f645372a700 1 -- 172.21.15.139:0/3727554342 shutdown_connections 2021-11-20T05:58:37.625 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.632+0000 7f645372a700 1 --2- 172.21.15.139:0/3727554342 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f644c0fa020 0x7f644c0fa440 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:37.625 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.632+0000 7f645372a700 1 -- 172.21.15.139:0/3727554342 >> 172.21.15.139:0/3727554342 conn(0x7f644c0f5a80 msgr2=0x7f644c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:37.625 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.632+0000 7f645372a700 1 -- 172.21.15.139:0/3727554342 shutdown_connections 2021-11-20T05:58:37.625 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.632+0000 7f645372a700 1 -- 172.21.15.139:0/3727554342 wait complete. 2021-11-20T05:58:37.626 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.633+0000 7f645372a700 1 Processor -- start 2021-11-20T05:58:37.626 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.633+0000 7f645372a700 1 -- start start 2021-11-20T05:58:37.626 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.633+0000 7f645372a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f644c0fa020 0x7f644c0fcc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:37.626 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.633+0000 7f645372a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c0fb090 con 0x7f644c0fa020 2021-11-20T05:58:37.626 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.633+0000 7f64514c6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f644c0fa020 0x7f644c0fcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:37.627 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.633+0000 7f64514c6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f644c0fa020 0x7f644c0fcc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56418/0 (socket says 172.21.15.139:56418) 2021-11-20T05:58:37.627 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.633+0000 7f64514c6700 1 -- 172.21.15.139:0/1114847273 learned_addr learned my addr 172.21.15.139:0/1114847273 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:37.627 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.633+0000 7f64514c6700 1 -- 172.21.15.139:0/1114847273 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644800b200 con 0x7f644c0fa020 2021-11-20T05:58:37.627 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.633+0000 7f64514c6700 1 --2- 172.21.15.139:0/1114847273 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f644c0fa020 0x7f644c0fcc10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f6448000c00 tx=0x7f6448006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:37.628 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.633+0000 7f6439ffb700 1 -- 172.21.15.139:0/1114847273 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6448011690 con 0x7f644c0fa020 2021-11-20T05:58:37.628 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.633+0000 7f6439ffb700 1 -- 172.21.15.139:0/1114847273 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f644800d9f0 con 0x7f644c0fa020 2021-11-20T05:58:37.628 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.633+0000 7f645372a700 1 -- 172.21.15.139:0/1114847273 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f644c0fb230 con 0x7f644c0fa020 2021-11-20T05:58:37.628 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.633+0000 7f6439ffb700 1 -- 172.21.15.139:0/1114847273 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f644801b710 con 0x7f644c0fa020 2021-11-20T05:58:37.628 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.634+0000 7f645372a700 1 -- 172.21.15.139:0/1114847273 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644c0fb640 con 0x7f644c0fa020 2021-11-20T05:58:37.629 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.634+0000 7f6439ffb700 1 -- 172.21.15.139:0/1114847273 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f644800f070 con 0x7f644c0fa020 2021-11-20T05:58:37.629 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.634+0000 7f6439ffb700 1 -- 172.21.15.139:0/1114847273 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6448025a30 con 0x7f644c0fa020 2021-11-20T05:58:37.629 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.635+0000 7f645372a700 1 -- 172.21.15.139:0/1114847273 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f644c045ad0 con 0x7f644c0fa020 2021-11-20T05:58:37.630 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.637+0000 7f6439ffb700 1 -- 172.21.15.139:0/1114847273 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f6448016070 con 0x7f644c0fa020 2021-11-20T05:58:37.688 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.700+0000 7f645372a700 1 -- 172.21.15.139:0/1114847273 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f644c027a00 con 0x7f644c0fa020 2021-11-20T05:58:37.689 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.703+0000 7f6439ffb700 1 -- 172.21.15.139:0/1114847273 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f644c027a00 con 0x7f644c0fa020 2021-11-20T05:58:37.690 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.704+0000 7f645372a700 1 -- 172.21.15.139:0/1114847273 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f644c0fa020 msgr2=0x7f644c0fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:37.691 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.704+0000 7f645372a700 1 --2- 172.21.15.139:0/1114847273 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f644c0fa020 0x7f644c0fcc10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f6448000c00 tx=0x7f6448006ee0).stop 2021-11-20T05:58:37.691 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.704+0000 7f645372a700 1 -- 172.21.15.139:0/1114847273 shutdown_connections 2021-11-20T05:58:37.691 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.704+0000 7f645372a700 1 --2- 172.21.15.139:0/1114847273 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f644c0fa020 0x7f644c0fcc10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:37.692 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.704+0000 7f645372a700 1 -- 172.21.15.139:0/1114847273 >> 172.21.15.139:0/1114847273 conn(0x7f644c0f5a80 msgr2=0x7f644c0f76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:37.692 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.704+0000 7f645372a700 1 -- 172.21.15.139:0/1114847273 shutdown_connections 2021-11-20T05:58:37.692 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:37.704+0000 7f645372a700 1 -- 172.21.15.139:0/1114847273 wait complete. 2021-11-20T05:58:38.056 INFO:teuthology.orchestra.run.smithi139.stderr:Wrote config to /etc/ceph/ceph.conf 2021-11-20T05:58:38.057 INFO:teuthology.orchestra.run.smithi139.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2021-11-20T05:58:38.057 INFO:teuthology.orchestra.run.smithi139.stderr:Creating mgr... 2021-11-20T05:58:38.057 INFO:teuthology.orchestra.run.smithi139.stderr:Verifying port 9283 ... 2021-11-20T05:58:38.059 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl daemon-reload 2021-11-20T05:58:38.145 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl stop ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mgr.smithi139.dplyxn 2021-11-20T05:58:38.166 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl reset-failed ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mgr.smithi139.dplyxn 2021-11-20T05:58:38.170 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Failed to reset failed state of unit ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mgr.smithi139.dplyxn.service: Unit ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mgr.smithi139.dplyxn.service not loaded. 2021-11-20T05:58:38.171 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl enable ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mgr.smithi139.dplyxn 2021-11-20T05:58:38.177 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: Created symlink /etc/systemd/system/ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c.target.wants/ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mgr.smithi139.dplyxn.service → /etc/systemd/system/ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@.service. 2021-11-20T05:58:38.257 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl start ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mgr.smithi139.dplyxn 2021-11-20T05:58:38.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:38 smithi139 conmon[33501]: audit 2021-11-20T05:58:37.703783+0000 mon.smithi139 (mon 2021-11-20T05:58:38.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:38 smithi139 conmon[33501]: .0) 6 : audit [INF] from='client.? 172.21.15.139:0/1114847273' entity='client.admin' 2021-11-20T05:58:38.785 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T05:58:38.793 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: disabled 2021-11-20T05:58:38.794 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T05:58:38.801 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: inactive 2021-11-20T05:58:38.802 INFO:teuthology.orchestra.run.smithi139.stderr:firewalld.service is not enabled 2021-11-20T05:58:38.802 INFO:teuthology.orchestra.run.smithi139.stderr:Not possible to enable service . firewalld.service is not available 2021-11-20T05:58:38.802 INFO:teuthology.orchestra.run.smithi139.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-11-20T05:58:38.803 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T05:58:38.809 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: disabled 2021-11-20T05:58:38.809 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T05:58:38.816 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: inactive 2021-11-20T05:58:38.817 INFO:teuthology.orchestra.run.smithi139.stderr:firewalld.service is not enabled 2021-11-20T05:58:38.817 INFO:teuthology.orchestra.run.smithi139.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2021-11-20T05:58:38.818 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for mgr to start... 2021-11-20T05:58:38.819 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for mgr... 2021-11-20T05:58:38.819 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-20T05:58:39.298 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.308+0000 7ff325a56700 1 Processor -- start 2021-11-20T05:58:39.298 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.308+0000 7ff325a56700 1 -- start start 2021-11-20T05:58:39.299 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.308+0000 7ff325a56700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff32006fc30 0x7ff320100c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:39.299 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.308+0000 7ff325a56700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3200fc620 con 0x7ff32006fc30 2021-11-20T05:58:39.299 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.308+0000 7ff31effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff32006fc30 0x7ff320100c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:39.300 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.309+0000 7ff31effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff32006fc30 0x7ff320100c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56424/0 (socket says 172.21.15.139:56424) 2021-11-20T05:58:39.300 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.309+0000 7ff31effd700 1 -- 172.21.15.139:0/749789311 learned_addr learned my addr 172.21.15.139:0/749789311 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:39.300 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.309+0000 7ff31effd700 1 -- 172.21.15.139:0/749789311 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff320101120 con 0x7ff32006fc30 2021-11-20T05:58:39.300 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.309+0000 7ff31effd700 1 --2- 172.21.15.139:0/749789311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff32006fc30 0x7ff320100c30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7ff31400b5f0 tx=0x7ff314006ee0).ready entity=mon.0 client_cookie=35b44c6ac9399251 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:39.300 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.309+0000 7ff31dffb700 1 -- 172.21.15.139:0/749789311 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff31400d600 con 0x7ff32006fc30 2021-11-20T05:58:39.301 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.309+0000 7ff31dffb700 1 -- 172.21.15.139:0/749789311 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff31400dbd0 con 0x7ff32006fc30 2021-11-20T05:58:39.301 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.309+0000 7ff31dffb700 1 -- 172.21.15.139:0/749789311 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff314011c00 con 0x7ff32006fc30 2021-11-20T05:58:39.301 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.310+0000 7ff325a56700 1 -- 172.21.15.139:0/749789311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff32006fc30 msgr2=0x7ff320100c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:39.301 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.310+0000 7ff325a56700 1 --2- 172.21.15.139:0/749789311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff32006fc30 0x7ff320100c30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7ff31400b5f0 tx=0x7ff314006ee0).stop 2021-11-20T05:58:39.302 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.310+0000 7ff325a56700 1 -- 172.21.15.139:0/749789311 shutdown_connections 2021-11-20T05:58:39.302 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.310+0000 7ff325a56700 1 --2- 172.21.15.139:0/749789311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff32006fc30 0x7ff320100c30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:39.302 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.310+0000 7ff325a56700 1 -- 172.21.15.139:0/749789311 >> 172.21.15.139:0/749789311 conn(0x7ff3200f6310 msgr2=0x7ff3200f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:39.302 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.310+0000 7ff325a56700 1 -- 172.21.15.139:0/749789311 shutdown_connections 2021-11-20T05:58:39.302 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.310+0000 7ff325a56700 1 -- 172.21.15.139:0/749789311 wait complete. 2021-11-20T05:58:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.311+0000 7ff325a56700 1 Processor -- start 2021-11-20T05:58:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.311+0000 7ff325a56700 1 -- start start 2021-11-20T05:58:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.311+0000 7ff325a56700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff32006fc30 0x7ff32010cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.311+0000 7ff325a56700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32010d050 con 0x7ff32006fc30 2021-11-20T05:58:39.303 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.311+0000 7ff31effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff32006fc30 0x7ff32010cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.311+0000 7ff31effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff32006fc30 0x7ff32010cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56426/0 (socket says 172.21.15.139:56426) 2021-11-20T05:58:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.311+0000 7ff31effd700 1 -- 172.21.15.139:0/3446471667 learned_addr learned my addr 172.21.15.139:0/3446471667 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.311+0000 7ff31effd700 1 -- 172.21.15.139:0/3446471667 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31400b200 con 0x7ff32006fc30 2021-11-20T05:58:39.304 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.312+0000 7ff31effd700 1 --2- 172.21.15.139:0/3446471667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff32006fc30 0x7ff32010cb60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff31400b190 tx=0x7ff314006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:39.305 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.312+0000 7ff3077fe700 1 -- 172.21.15.139:0/3446471667 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff31401b410 con 0x7ff32006fc30 2021-11-20T05:58:39.306 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.312+0000 7ff3077fe700 1 -- 172.21.15.139:0/3446471667 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff31401b9e0 con 0x7ff32006fc30 2021-11-20T05:58:39.306 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.312+0000 7ff325a56700 1 -- 172.21.15.139:0/3446471667 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32010d250 con 0x7ff32006fc30 2021-11-20T05:58:39.306 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.312+0000 7ff3077fe700 1 -- 172.21.15.139:0/3446471667 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff31401a880 con 0x7ff32006fc30 2021-11-20T05:58:39.307 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.312+0000 7ff325a56700 1 -- 172.21.15.139:0/3446471667 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32010d660 con 0x7ff32006fc30 2021-11-20T05:58:39.307 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.312+0000 7ff3077fe700 1 -- 172.21.15.139:0/3446471667 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7ff31400f070 con 0x7ff32006fc30 2021-11-20T05:58:39.307 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.312+0000 7ff3077fe700 1 -- 172.21.15.139:0/3446471667 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff314025c60 con 0x7ff32006fc30 2021-11-20T05:58:39.307 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.313+0000 7ff325a56700 1 -- 172.21.15.139:0/3446471667 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff320045ad0 con 0x7ff32006fc30 2021-11-20T05:58:39.308 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.316+0000 7ff3077fe700 1 -- 172.21.15.139:0/3446471667 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7ff314016070 con 0x7ff32006fc30 2021-11-20T05:58:39.382 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.384+0000 7ff325a56700 1 -- 172.21.15.139:0/3446471667 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff320106a00 con 0x7ff32006fc30 2021-11-20T05:58:39.383 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.384+0000 7ff3077fe700 1 -- 172.21.15.139:0/3446471667 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7ff31401bb50 con 0x7ff32006fc30 2021-11-20T05:58:39.383 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.383 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: { 2021-11-20T05:58:39.384 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "fsid": "d819f2cc-49c6-11ec-8c2c-001a4aab830c", 2021-11-20T05:58:39.384 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "health": { 2021-11-20T05:58:39.384 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-20T05:58:39.384 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "checks": {}, 2021-11-20T05:58:39.385 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mutes": [] 2021-11-20T05:58:39.385 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:39.385 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-20T05:58:39.385 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum": [ 2021-11-20T05:58:39.385 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 0 2021-11-20T05:58:39.386 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2021-11-20T05:58:39.386 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-20T05:58:39.386 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "smithi139" 2021-11-20T05:58:39.386 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2021-11-20T05:58:39.387 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum_age": 2, 2021-11-20T05:58:39.387 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "monmap": { 2021-11-20T05:58:39.388 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:58:39.388 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-20T05:58:39.388 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-20T05:58:39.389 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:39.389 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osdmap": { 2021-11-20T05:58:39.390 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:58:39.390 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-20T05:58:39.391 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-20T05:58:39.391 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-20T05:58:39.391 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-20T05:58:39.392 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-20T05:58:39.392 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-20T05:58:39.392 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:39.392 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "pgmap": { 2021-11-20T05:58:39.393 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-20T05:58:39.393 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-20T05:58:39.393 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-20T05:58:39.393 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-20T05:58:39.393 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-20T05:58:39.394 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-20T05:58:39.394 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-20T05:58:39.394 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-20T05:58:39.394 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:39.395 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "fsmap": { 2021-11-20T05:58:39.395 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:58:39.395 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-20T05:58:39.395 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-20T05:58:39.396 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:39.396 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-20T05:58:39.396 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "available": false, 2021-11-20T05:58:39.396 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-20T05:58:39.397 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "modules": [ 2021-11-20T05:58:39.397 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "iostat", 2021-11-20T05:58:39.397 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "restful" 2021-11-20T05:58:39.397 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2021-11-20T05:58:39.397 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "services": {} 2021-11-20T05:58:39.398 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:39.398 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "servicemap": { 2021-11-20T05:58:39.398 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:58:39.398 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "modified": "2021-11-20T05:58:33.540798+0000", 2021-11-20T05:58:39.399 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "services": {} 2021-11-20T05:58:39.399 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:39.399 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-20T05:58:39.399 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: } 2021-11-20T05:58:39.399 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.386+0000 7ff325a56700 1 -- 172.21.15.139:0/3446471667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff32006fc30 msgr2=0x7ff32010cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:39.400 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.386+0000 7ff325a56700 1 --2- 172.21.15.139:0/3446471667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff32006fc30 0x7ff32010cb60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff31400b190 tx=0x7ff314006ee0).stop 2021-11-20T05:58:39.400 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.386+0000 7ff325a56700 1 -- 172.21.15.139:0/3446471667 shutdown_connections 2021-11-20T05:58:39.400 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.386+0000 7ff325a56700 1 --2- 172.21.15.139:0/3446471667 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff32006fc30 0x7ff32010cb60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:39.400 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.386+0000 7ff325a56700 1 -- 172.21.15.139:0/3446471667 >> 172.21.15.139:0/3446471667 conn(0x7ff3200f6310 msgr2=0x7ff3200f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:39.401 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.386+0000 7ff325a56700 1 -- 172.21.15.139:0/3446471667 shutdown_connections 2021-11-20T05:58:39.401 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:39.386+0000 7ff325a56700 1 -- 172.21.15.139:0/3446471667 wait complete. 2021-11-20T05:58:39.667 INFO:teuthology.orchestra.run.smithi139.stderr:mgr not available, waiting (1/15)... 2021-11-20T05:58:40.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:39 smithi139 conmon[33501]: audit 2021-11-20T05:58:39.385338+0000 mon.smithi139 ( 2021-11-20T05:58:40.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:39 smithi139 conmon[33501]: mon.0) 7 : audit [DBG] from='client.? 172.21.15.139:0/3446471667' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-20T05:58:41.670 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-20T05:58:42.128 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.139+0000 7f19a4ce0700 1 Processor -- start 2021-11-20T05:58:42.128 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.139+0000 7f19a4ce0700 1 -- start start 2021-11-20T05:58:42.128 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.140+0000 7f19a4ce0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19a00faad0 0x7f19a00faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:42.129 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.140+0000 7f19a4ce0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a00fb3e0 con 0x7f19a00faad0 2021-11-20T05:58:42.129 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.140+0000 7f199e59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19a00faad0 0x7f19a00faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:42.129 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.140+0000 7f199e59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19a00faad0 0x7f19a00faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56428/0 (socket says 172.21.15.139:56428) 2021-11-20T05:58:42.129 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.140+0000 7f199e59c700 1 -- 172.21.15.139:0/3257562528 learned_addr learned my addr 172.21.15.139:0/3257562528 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:42.129 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.140+0000 7f199e59c700 1 -- 172.21.15.139:0/3257562528 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a00fb520 con 0x7f19a00faad0 2021-11-20T05:58:42.130 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.140+0000 7f199e59c700 1 --2- 172.21.15.139:0/3257562528 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19a00faad0 0x7f19a00faef0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f199400b5f0 tx=0x7f1994006ee0).ready entity=mon.0 client_cookie=21e0fcb16c584436 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:42.130 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.141+0000 7f199d59a700 1 -- 172.21.15.139:0/3257562528 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f199400d600 con 0x7f19a00faad0 2021-11-20T05:58:42.130 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.141+0000 7f199d59a700 1 -- 172.21.15.139:0/3257562528 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f199400dbd0 con 0x7f19a00faad0 2021-11-20T05:58:42.130 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.141+0000 7f199d59a700 1 -- 172.21.15.139:0/3257562528 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1994011c00 con 0x7f19a00faad0 2021-11-20T05:58:42.131 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.141+0000 7f19a4ce0700 1 -- 172.21.15.139:0/3257562528 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19a00faad0 msgr2=0x7f19a00faef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:42.131 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.141+0000 7f19a4ce0700 1 --2- 172.21.15.139:0/3257562528 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19a00faad0 0x7f19a00faef0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f199400b5f0 tx=0x7f1994006ee0).stop 2021-11-20T05:58:42.131 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.141+0000 7f19a4ce0700 1 -- 172.21.15.139:0/3257562528 shutdown_connections 2021-11-20T05:58:42.132 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.141+0000 7f19a4ce0700 1 --2- 172.21.15.139:0/3257562528 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19a00faad0 0x7f19a00faef0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:42.133 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.141+0000 7f19a4ce0700 1 -- 172.21.15.139:0/3257562528 >> 172.21.15.139:0/3257562528 conn(0x7f19a00f6310 msgr2=0x7f19a00f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:42.133 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.141+0000 7f19a4ce0700 1 -- 172.21.15.139:0/3257562528 shutdown_connections 2021-11-20T05:58:42.133 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.141+0000 7f19a4ce0700 1 -- 172.21.15.139:0/3257562528 wait complete. 2021-11-20T05:58:42.133 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.142+0000 7f19a4ce0700 1 Processor -- start 2021-11-20T05:58:42.134 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.142+0000 7f19a4ce0700 1 -- start start 2021-11-20T05:58:42.134 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.142+0000 7f19a4ce0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19a00faad0 0x7f19a010d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:42.134 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.142+0000 7f19a4ce0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a006fc70 con 0x7f19a00faad0 2021-11-20T05:58:42.135 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.142+0000 7f199e59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19a00faad0 0x7f19a010d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:42.135 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.142+0000 7f199e59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19a00faad0 0x7f19a010d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56430/0 (socket says 172.21.15.139:56430) 2021-11-20T05:58:42.135 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.142+0000 7f199e59c700 1 -- 172.21.15.139:0/2589494023 learned_addr learned my addr 172.21.15.139:0/2589494023 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:42.135 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.143+0000 7f199e59c700 1 -- 172.21.15.139:0/2589494023 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199400b200 con 0x7f19a00faad0 2021-11-20T05:58:42.136 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.143+0000 7f199e59c700 1 --2- 172.21.15.139:0/2589494023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19a00faad0 0x7f19a010d6a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f199400e040 tx=0x7f1994006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:42.136 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.143+0000 7f198effd700 1 -- 172.21.15.139:0/2589494023 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f199401b410 con 0x7f19a00faad0 2021-11-20T05:58:42.136 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.143+0000 7f198effd700 1 -- 172.21.15.139:0/2589494023 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f199401b9e0 con 0x7f19a00faad0 2021-11-20T05:58:42.136 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.143+0000 7f198effd700 1 -- 172.21.15.139:0/2589494023 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f199401a820 con 0x7f19a00faad0 2021-11-20T05:58:42.137 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.143+0000 7f19a4ce0700 1 -- 172.21.15.139:0/2589494023 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a006fe70 con 0x7f19a00faad0 2021-11-20T05:58:42.137 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.143+0000 7f19a4ce0700 1 -- 172.21.15.139:0/2589494023 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a010dd50 con 0x7f19a00faad0 2021-11-20T05:58:42.137 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.143+0000 7f198effd700 1 -- 172.21.15.139:0/2589494023 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f199400f070 con 0x7f19a00faad0 2021-11-20T05:58:42.138 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.143+0000 7f198effd700 1 -- 172.21.15.139:0/2589494023 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1994025bc0 con 0x7f19a00faad0 2021-11-20T05:58:42.138 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.144+0000 7f19a4ce0700 1 -- 172.21.15.139:0/2589494023 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a0045ad0 con 0x7f19a00faad0 2021-11-20T05:58:42.138 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.146+0000 7f198effd700 1 -- 172.21.15.139:0/2589494023 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1994016070 con 0x7f19a00faad0 2021-11-20T05:58:42.204 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.215+0000 7f19a4ce0700 1 -- 172.21.15.139:0/2589494023 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f19a00586e0 con 0x7f19a00faad0 2021-11-20T05:58:42.204 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.216+0000 7f198effd700 1 -- 172.21.15.139:0/2589494023 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f1994025e50 con 0x7f19a00faad0 2021-11-20T05:58:42.204 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.205 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: { 2021-11-20T05:58:42.205 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "fsid": "d819f2cc-49c6-11ec-8c2c-001a4aab830c", 2021-11-20T05:58:42.205 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "health": { 2021-11-20T05:58:42.205 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-20T05:58:42.205 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "checks": {}, 2021-11-20T05:58:42.206 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mutes": [] 2021-11-20T05:58:42.206 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:42.206 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-20T05:58:42.206 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum": [ 2021-11-20T05:58:42.207 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 0 2021-11-20T05:58:42.207 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2021-11-20T05:58:42.207 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-20T05:58:42.209 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "smithi139" 2021-11-20T05:58:42.210 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2021-11-20T05:58:42.210 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum_age": 5, 2021-11-20T05:58:42.210 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "monmap": { 2021-11-20T05:58:42.210 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:58:42.211 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-20T05:58:42.211 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-20T05:58:42.211 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:42.211 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osdmap": { 2021-11-20T05:58:42.211 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:58:42.212 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-20T05:58:42.212 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-20T05:58:42.212 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-20T05:58:42.212 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-20T05:58:42.213 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-20T05:58:42.213 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-20T05:58:42.213 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:42.213 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "pgmap": { 2021-11-20T05:58:42.213 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-20T05:58:42.214 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-20T05:58:42.214 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-20T05:58:42.214 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-20T05:58:42.214 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-20T05:58:42.215 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-20T05:58:42.215 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-20T05:58:42.215 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-20T05:58:42.215 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:42.215 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "fsmap": { 2021-11-20T05:58:42.216 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:58:42.216 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-20T05:58:42.216 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-20T05:58:42.216 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:42.217 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-20T05:58:42.217 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "available": false, 2021-11-20T05:58:42.217 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-20T05:58:42.217 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "modules": [ 2021-11-20T05:58:42.217 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "iostat", 2021-11-20T05:58:42.218 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "restful" 2021-11-20T05:58:42.218 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2021-11-20T05:58:42.218 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "services": {} 2021-11-20T05:58:42.218 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:42.218 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "servicemap": { 2021-11-20T05:58:42.219 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:58:42.219 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "modified": "2021-11-20T05:58:33.540798+0000", 2021-11-20T05:58:42.219 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "services": {} 2021-11-20T05:58:42.219 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:42.220 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-20T05:58:42.220 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: } 2021-11-20T05:58:42.220 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.217+0000 7f19a4ce0700 1 -- 172.21.15.139:0/2589494023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19a00faad0 msgr2=0x7f19a010d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:42.220 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.217+0000 7f19a4ce0700 1 --2- 172.21.15.139:0/2589494023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19a00faad0 0x7f19a010d6a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f199400e040 tx=0x7f1994006ee0).stop 2021-11-20T05:58:42.221 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.217+0000 7f19a4ce0700 1 -- 172.21.15.139:0/2589494023 shutdown_connections 2021-11-20T05:58:42.221 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.217+0000 7f19a4ce0700 1 --2- 172.21.15.139:0/2589494023 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19a00faad0 0x7f19a010d6a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:42.221 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.217+0000 7f19a4ce0700 1 -- 172.21.15.139:0/2589494023 >> 172.21.15.139:0/2589494023 conn(0x7f19a00f6310 msgr2=0x7f19a00f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:42.221 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.217+0000 7f19a4ce0700 1 -- 172.21.15.139:0/2589494023 shutdown_connections 2021-11-20T05:58:42.222 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:42.217+0000 7f19a4ce0700 1 -- 172.21.15.139:0/2589494023 wait complete. 2021-11-20T05:58:42.606 INFO:teuthology.orchestra.run.smithi139.stderr:mgr not available, waiting (2/15)... 2021-11-20T05:58:43.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:43 smithi139 conmon[33501]: audit 2021-11-20T05:58:42.216711+0000 mon.smithi139 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.139:0/2589494023' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-20T05:58:44.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: cluster 2021-11-20T05:58:43.680695+0000 mon.smithi139 (mon.0 2021-11-20T05:58:44.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: ) 9 : cluster [INF] Activating manager daemon smithi139.dplyxn 2021-11-20T05:58:44.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: cluster 2021-11-20T05:58:43.683197+0000 mon.smithi139 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi139.dplyxn(active, starting, since 0.0026146s) 2021-11-20T05:58:44.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: audit 2021-11-20T05:58:43.683962+0000 mon.smithi139 (mon.0) 11 : 2021-11-20T05:58:44.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: audit [DBG] from='mgr.14100 172.21.15.139:0/4176409003' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T05:58:44.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: audit 2021-11-20T05: 2021-11-20T05:58:44.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: 58:43.684101+0000 mon.smithi139 (mon.0) 12 : audit [DBG] 2021-11-20T05:58:44.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: from='mgr.14100 172.21.15.139:0/4176409003' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T05:58:44.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: audit 2021-11-20T05:58 2021-11-20T05:58:44.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: :43.684210+0000 mon.smithi139 (mon.0) 13 2021-11-20T05:58:44.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: : audit [DBG] from='mgr.14100 172.21.15.139:0/4176409003' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T05:58:44.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T05:58:44.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: 20T05:58:43.684353+0000 mon.smithi139 (mon 2021-11-20T05:58:44.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: .0) 14 : audit [DBG] from='mgr.14100 172.21.15.139:0/4176409003' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi139"}]: dispatch 2021-11-20T05:58:44.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: audit 2021- 2021-11-20T05:58:44.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: 11-20T05:58:43.684521+0000 mon.smithi139 2021-11-20T05:58:44.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.139:0/4176409003' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mgr metadata", "who": "smithi139.dplyxn", "id": "smithi139.dplyxn"}]: dispatch 2021-11-20T05:58:44.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: cluster 2021-11-20T05:58:44.534 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: 2021-11-20T05:58:43.697834+0000 2021-11-20T05:58:44.534 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: mon.smithi139 (mon.0) 16 : cluster [INF] Manager daemon smithi139.dplyxn is now available 2021-11-20T05:58:44.534 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: 2021-11-20T05:58:44.534 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: audit 2021-11-20T05:58: 2021-11-20T05:58:44.535 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: 43.705499+0000 mon.smithi139 (mon.0) 17 2021-11-20T05:58:44.535 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: : audit [INF] from='mgr.14100 172.21.15.139:0/4176409003' entity='mgr.smithi139.dplyxn' 2021-11-20T05:58:44.535 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T05:58:44.535 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: 20T05:58:43.709641+0000 mon.smithi139 (mon 2021-11-20T05:58:44.535 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: .0) 18 : audit [INF] from='mgr.14100 172.21.15.139:0/4176409003' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:58:44.536 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: audit 2021-11-20T05:58:44.536 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: 2021-11-20T05:58:43.710708+0000 2021-11-20T05:58:44.536 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: mon.smithi139 (mon.0) 19 : audit [INF] 2021-11-20T05:58:44.536 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: from='mgr.14100 172.21.15.139:0/4176409003' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T05:58:44.536 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: audit 2021-11-20T05:58:43 2021-11-20T05:58:44.537 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: .715621+0000 mon.smithi139 (mon.0) 20 : 2021-11-20T05:58:44.537 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: audit [INF] from='mgr.14100 172.21.15.139:0/4176409003' entity='mgr.smithi139.dplyxn' 2021-11-20T05:58:44.537 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: audit 2021-11-20 2021-11-20T05:58:44.537 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: T05:58:43.717276+0000 mon.smithi139 (mon 2021-11-20T05:58:44.538 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:44 smithi139 conmon[33501]: .0) 21 : audit [INF] from='mgr.14100 172.21.15.139:0/4176409003' entity='mgr.smithi139.dplyxn' 2021-11-20T05:58:44.609 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-20T05:58:45.093 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.101+0000 7f1126028700 1 Processor -- start 2021-11-20T05:58:45.093 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.101+0000 7f1126028700 1 -- start start 2021-11-20T05:58:45.093 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.101+0000 7f1126028700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f112006fbf0 0x7f11200fe690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:45.094 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.101+0000 7f1126028700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11200feb80 con 0x7f112006fbf0 2021-11-20T05:58:45.094 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.101+0000 7f1125026700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f112006fbf0 0x7f11200fe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:45.094 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.101+0000 7f1125026700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f112006fbf0 0x7f11200fe690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56450/0 (socket says 172.21.15.139:56450) 2021-11-20T05:58:45.094 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.101+0000 7f1125026700 1 -- 172.21.15.139:0/1329482636 learned_addr learned my addr 172.21.15.139:0/1329482636 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:45.094 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.102+0000 7f1125026700 1 -- 172.21.15.139:0/1329482636 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11200fecc0 con 0x7f112006fbf0 2021-11-20T05:58:45.095 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.102+0000 7f1125026700 1 --2- 172.21.15.139:0/1329482636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f112006fbf0 0x7f11200fe690 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f111c00b5f0 tx=0x7f111c006ee0).ready entity=mon.0 client_cookie=d2ed5ecff02540f9 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:45.095 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.102+0000 7f110ffff700 1 -- 172.21.15.139:0/1329482636 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f111c00d600 con 0x7f112006fbf0 2021-11-20T05:58:45.095 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.102+0000 7f110ffff700 1 -- 172.21.15.139:0/1329482636 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f111c00dbd0 con 0x7f112006fbf0 2021-11-20T05:58:45.095 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.102+0000 7f110ffff700 1 -- 172.21.15.139:0/1329482636 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f111c011c30 con 0x7f112006fbf0 2021-11-20T05:58:45.096 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.103+0000 7f1126028700 1 -- 172.21.15.139:0/1329482636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f112006fbf0 msgr2=0x7f11200fe690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:45.096 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.103+0000 7f1126028700 1 --2- 172.21.15.139:0/1329482636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f112006fbf0 0x7f11200fe690 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f111c00b5f0 tx=0x7f111c006ee0).stop 2021-11-20T05:58:45.096 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.103+0000 7f1126028700 1 -- 172.21.15.139:0/1329482636 shutdown_connections 2021-11-20T05:58:45.096 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.103+0000 7f1126028700 1 --2- 172.21.15.139:0/1329482636 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f112006fbf0 0x7f11200fe690 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:45.096 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.103+0000 7f1126028700 1 -- 172.21.15.139:0/1329482636 >> 172.21.15.139:0/1329482636 conn(0x7f11200f6110 msgr2=0x7f11200f8530 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:45.097 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.103+0000 7f1126028700 1 -- 172.21.15.139:0/1329482636 shutdown_connections 2021-11-20T05:58:45.097 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.103+0000 7f1126028700 1 -- 172.21.15.139:0/1329482636 wait complete. 2021-11-20T05:58:45.097 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.103+0000 7f1126028700 1 Processor -- start 2021-11-20T05:58:45.097 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.103+0000 7f1126028700 1 -- start start 2021-11-20T05:58:45.097 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.104+0000 7f1126028700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f112006fbf0 0x7f112010a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:45.098 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.104+0000 7f1126028700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112010ac80 con 0x7f112006fbf0 2021-11-20T05:58:45.098 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.104+0000 7f1125026700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f112006fbf0 0x7f112010a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:45.098 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.104+0000 7f1125026700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f112006fbf0 0x7f112010a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56452/0 (socket says 172.21.15.139:56452) 2021-11-20T05:58:45.098 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.104+0000 7f1125026700 1 -- 172.21.15.139:0/1609443372 learned_addr learned my addr 172.21.15.139:0/1609443372 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:45.099 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.104+0000 7f1125026700 1 -- 172.21.15.139:0/1609443372 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c00b200 con 0x7f112006fbf0 2021-11-20T05:58:45.099 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.104+0000 7f1125026700 1 --2- 172.21.15.139:0/1609443372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f112006fbf0 0x7f112010a790 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f111c011850 tx=0x7f111c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:45.099 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.104+0000 7f110dffb700 1 -- 172.21.15.139:0/1609443372 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f111c01b570 con 0x7f112006fbf0 2021-11-20T05:58:45.099 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.104+0000 7f110dffb700 1 -- 172.21.15.139:0/1609443372 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f111c01bb40 con 0x7f112006fbf0 2021-11-20T05:58:45.099 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.104+0000 7f110dffb700 1 -- 172.21.15.139:0/1609443372 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f111c01aa10 con 0x7f112006fbf0 2021-11-20T05:58:45.100 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.104+0000 7f1126028700 1 -- 172.21.15.139:0/1609443372 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f112010ae80 con 0x7f112006fbf0 2021-11-20T05:58:45.100 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.105+0000 7f1126028700 1 -- 172.21.15.139:0/1609443372 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f112010b290 con 0x7f112006fbf0 2021-11-20T05:58:45.100 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.105+0000 7f110dffb700 1 -- 172.21.15.139:0/1609443372 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f111c00f070 con 0x7f112006fbf0 2021-11-20T05:58:45.100 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.105+0000 7f110dffb700 1 --2- 172.21.15.139:0/1609443372 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f110802fcc0 0x7f1108032180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:45.100 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.106+0000 7f110dffb700 1 -- 172.21.15.139:0/1609443372 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f111c040ff0 con 0x7f112006fbf0 2021-11-20T05:58:45.101 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.106+0000 7f1124825700 1 --2- 172.21.15.139:0/1609443372 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f110802fcc0 0x7f1108032180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:45.101 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.106+0000 7f1126028700 1 -- 172.21.15.139:0/1609443372 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1120059100 con 0x7f112006fbf0 2021-11-20T05:58:45.102 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.106+0000 7f1124825700 1 --2- 172.21.15.139:0/1609443372 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f110802fcc0 0x7f1108032180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f11100096a0 tx=0x7f1110006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:45.102 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.110+0000 7f110dffb700 1 -- 172.21.15.139:0/1609443372 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f111c016080 con 0x7f112006fbf0 2021-11-20T05:58:45.307 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.312+0000 7f1126028700 1 -- 172.21.15.139:0/1609443372 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1120045ad0 con 0x7f112006fbf0 2021-11-20T05:58:45.307 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.312+0000 7f110dffb700 1 -- 172.21.15.139:0/1609443372 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7f111c024410 con 0x7f112006fbf0 2021-11-20T05:58:45.307 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.308 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: { 2021-11-20T05:58:45.308 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "fsid": "d819f2cc-49c6-11ec-8c2c-001a4aab830c", 2021-11-20T05:58:45.308 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "health": { 2021-11-20T05:58:45.308 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-20T05:58:45.309 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "checks": {}, 2021-11-20T05:58:45.309 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mutes": [] 2021-11-20T05:58:45.309 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:45.309 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-20T05:58:45.309 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum": [ 2021-11-20T05:58:45.310 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 0 2021-11-20T05:58:45.310 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2021-11-20T05:58:45.310 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-20T05:58:45.310 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "smithi139" 2021-11-20T05:58:45.310 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2021-11-20T05:58:45.311 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "quorum_age": 8, 2021-11-20T05:58:45.311 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "monmap": { 2021-11-20T05:58:45.311 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:58:45.311 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-20T05:58:45.312 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-20T05:58:45.312 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:45.312 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osdmap": { 2021-11-20T05:58:45.312 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:58:45.312 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-20T05:58:45.313 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-20T05:58:45.313 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-20T05:58:45.313 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-20T05:58:45.313 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-20T05:58:45.314 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-20T05:58:45.314 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:45.314 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "pgmap": { 2021-11-20T05:58:45.314 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-20T05:58:45.314 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-20T05:58:45.315 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-20T05:58:45.315 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-20T05:58:45.315 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-20T05:58:45.315 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-20T05:58:45.315 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-20T05:58:45.316 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-20T05:58:45.317 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:45.317 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "fsmap": { 2021-11-20T05:58:45.317 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:58:45.317 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-20T05:58:45.318 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-20T05:58:45.318 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:45.318 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-20T05:58:45.318 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "available": true, 2021-11-20T05:58:45.318 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-20T05:58:45.319 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "modules": [ 2021-11-20T05:58:45.319 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "iostat", 2021-11-20T05:58:45.319 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "restful" 2021-11-20T05:58:45.319 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ], 2021-11-20T05:58:45.320 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "services": {} 2021-11-20T05:58:45.320 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:45.320 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "servicemap": { 2021-11-20T05:58:45.321 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-20T05:58:45.321 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "modified": "2021-11-20T05:58:33.540798+0000", 2021-11-20T05:58:45.321 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "services": {} 2021-11-20T05:58:45.322 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: }, 2021-11-20T05:58:45.322 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-20T05:58:45.322 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: } 2021-11-20T05:58:45.322 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.314+0000 7f1126028700 1 -- 172.21.15.139:0/1609443372 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f110802fcc0 msgr2=0x7f1108032180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:45.323 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.314+0000 7f1126028700 1 --2- 172.21.15.139:0/1609443372 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f110802fcc0 0x7f1108032180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f11100096a0 tx=0x7f1110006b40).stop 2021-11-20T05:58:45.323 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.314+0000 7f1126028700 1 -- 172.21.15.139:0/1609443372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f112006fbf0 msgr2=0x7f112010a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:45.323 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.314+0000 7f1126028700 1 --2- 172.21.15.139:0/1609443372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f112006fbf0 0x7f112010a790 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f111c011850 tx=0x7f111c006ee0).stop 2021-11-20T05:58:45.323 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.314+0000 7f1126028700 1 -- 172.21.15.139:0/1609443372 shutdown_connections 2021-11-20T05:58:45.324 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.314+0000 7f1126028700 1 --2- 172.21.15.139:0/1609443372 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f110802fcc0 0x7f1108032180 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:45.324 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.314+0000 7f1126028700 1 --2- 172.21.15.139:0/1609443372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f112006fbf0 0x7f112010a790 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:45.324 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.314+0000 7f1126028700 1 -- 172.21.15.139:0/1609443372 >> 172.21.15.139:0/1609443372 conn(0x7f11200f6110 msgr2=0x7f11200f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:45.324 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.314+0000 7f1126028700 1 -- 172.21.15.139:0/1609443372 shutdown_connections 2021-11-20T05:58:45.324 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:45.314+0000 7f1126028700 1 -- 172.21.15.139:0/1609443372 wait complete. 2021-11-20T05:58:45.647 INFO:teuthology.orchestra.run.smithi139.stderr:mgr is available 2021-11-20T05:58:45.648 INFO:teuthology.orchestra.run.smithi139.stderr:Enabling cephadm module... 2021-11-20T05:58:45.648 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2021-11-20T05:58:45.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:45 smithi139 conmon[33501]: cluster 2021-11-20T05:58:44.685929+0000 mon.smithi139 (mon.0) 22 2021-11-20T05:58:45.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:45 smithi139 conmon[33501]: : cluster [DBG] mgrmap e3: smithi139.dplyxn(active, since 1.00535s) 2021-11-20T05:58:45.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:45 smithi139 conmon[33501]: audit 2021-11-20T05:58:45.313407+0000 mon.smithi139 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.139:0/1609443372' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-20T05:58:46.089 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.100+0000 7fe2e3036700 1 Processor -- start 2021-11-20T05:58:46.089 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.100+0000 7fe2e3036700 1 -- start start 2021-11-20T05:58:46.089 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.100+0000 7fe2e3036700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe2dc0fd2a0 0x7fe2dc0fd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:46.089 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.100+0000 7fe2e3036700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc0fdbb0 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.090 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.100+0000 7fe2e0dd2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe2dc0fd2a0 0x7fe2dc0fd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:46.091 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.100+0000 7fe2e0dd2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe2dc0fd2a0 0x7fe2dc0fd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56456/0 (socket says 172.21.15.139:56456) 2021-11-20T05:58:46.091 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.100+0000 7fe2e0dd2700 1 -- 172.21.15.139:0/2224140955 learned_addr learned my addr 172.21.15.139:0/2224140955 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:46.091 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.100+0000 7fe2e0dd2700 1 -- 172.21.15.139:0/2224140955 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc0fdcf0 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.091 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.101+0000 7fe2e0dd2700 1 --2- 172.21.15.139:0/2224140955 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe2dc0fd2a0 0x7fe2dc0fd6c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fe2d000b5f0 tx=0x7fe2d0006ee0).ready entity=mon.0 client_cookie=b91fa104b28ce587 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:46.092 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.101+0000 7fe2db7fe700 1 -- 172.21.15.139:0/2224140955 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2d000d600 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.092 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.101+0000 7fe2db7fe700 1 -- 172.21.15.139:0/2224140955 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe2d000dbd0 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.092 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.101+0000 7fe2db7fe700 1 -- 172.21.15.139:0/2224140955 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2d0011c00 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.092 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.101+0000 7fe2e3036700 1 -- 172.21.15.139:0/2224140955 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe2dc0fd2a0 msgr2=0x7fe2dc0fd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:46.093 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.101+0000 7fe2e3036700 1 --2- 172.21.15.139:0/2224140955 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe2dc0fd2a0 0x7fe2dc0fd6c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fe2d000b5f0 tx=0x7fe2d0006ee0).stop 2021-11-20T05:58:46.093 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.101+0000 7fe2e3036700 1 -- 172.21.15.139:0/2224140955 shutdown_connections 2021-11-20T05:58:46.093 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.101+0000 7fe2e3036700 1 --2- 172.21.15.139:0/2224140955 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe2dc0fd2a0 0x7fe2dc0fd6c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:46.093 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.101+0000 7fe2e3036700 1 -- 172.21.15.139:0/2224140955 >> 172.21.15.139:0/2224140955 conn(0x7fe2dc0f5a80 msgr2=0x7fe2dc0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:46.093 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.102+0000 7fe2e3036700 1 -- 172.21.15.139:0/2224140955 shutdown_connections 2021-11-20T05:58:46.094 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.102+0000 7fe2e3036700 1 -- 172.21.15.139:0/2224140955 wait complete. 2021-11-20T05:58:46.094 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.102+0000 7fe2e3036700 1 Processor -- start 2021-11-20T05:58:46.094 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.102+0000 7fe2e3036700 1 -- start start 2021-11-20T05:58:46.097 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.103+0000 7fe2e3036700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe2dc0fd2a0 0x7fe2dc10a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:46.097 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.103+0000 7fe2e3036700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc10aed0 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.097 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.103+0000 7fe2e0dd2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe2dc0fd2a0 0x7fe2dc10a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:46.097 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.103+0000 7fe2e0dd2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe2dc0fd2a0 0x7fe2dc10a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56458/0 (socket says 172.21.15.139:56458) 2021-11-20T05:58:46.098 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.103+0000 7fe2e0dd2700 1 -- 172.21.15.139:0/711825103 learned_addr learned my addr 172.21.15.139:0/711825103 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:46.098 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.103+0000 7fe2e0dd2700 1 -- 172.21.15.139:0/711825103 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d000b200 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.098 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.103+0000 7fe2e0dd2700 1 --2- 172.21.15.139:0/711825103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe2dc0fd2a0 0x7fe2dc10a9e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fe2d000b1d0 tx=0x7fe2d0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:46.098 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.103+0000 7fe2d97fa700 1 -- 172.21.15.139:0/711825103 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2d0011690 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.099 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.103+0000 7fe2d97fa700 1 -- 172.21.15.139:0/711825103 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe2d001b880 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.099 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.103+0000 7fe2e3036700 1 -- 172.21.15.139:0/711825103 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2dc10b0d0 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.099 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.103+0000 7fe2d97fa700 1 -- 172.21.15.139:0/711825103 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2d001a650 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.099 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.103+0000 7fe2e3036700 1 -- 172.21.15.139:0/711825103 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2dc10b4e0 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.099 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.104+0000 7fe2d97fa700 1 -- 172.21.15.139:0/711825103 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7fe2d000f070 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.100 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.104+0000 7fe2e3036700 1 -- 172.21.15.139:0/711825103 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2dc045ad0 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.100 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.104+0000 7fe2d97fa700 1 --2- 172.21.15.139:0/711825103 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7fe2c002fcc0 0x7fe2c0032180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:46.100 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.104+0000 7fe2d97fa700 1 -- 172.21.15.139:0/711825103 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe2d001b620 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.100 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.104+0000 7fe2dbfff700 1 --2- 172.21.15.139:0/711825103 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7fe2c002fcc0 0x7fe2c0032180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:46.101 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.105+0000 7fe2dbfff700 1 --2- 172.21.15.139:0/711825103 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7fe2c002fcc0 0x7fe2c0032180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe2c80096a0 tx=0x7fe2c8006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:46.101 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.108+0000 7fe2d97fa700 1 -- 172.21.15.139:0/711825103 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2d0022900 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.282 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.295+0000 7fe2e3036700 1 -- 172.21.15.139:0/711825103 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fe2dc0fa660 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.671 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.684+0000 7fe2d97fa700 1 -- 172.21.15.139:0/711825103 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe2d001b9f0 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.672 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.685+0000 7fe2d97fa700 1 -- 172.21.15.139:0/711825103 <== mon.0 v2:172.21.15.139:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v4) v1 ==== 86+0+0 (secure 0 0 0) 0x7fe2d0022310 con 0x7fe2dc0fd2a0 2021-11-20T05:58:46.675 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.687+0000 7fe2e3036700 1 -- 172.21.15.139:0/711825103 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7fe2c002fcc0 msgr2=0x7fe2c0032180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:46.675 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.687+0000 7fe2e3036700 1 --2- 172.21.15.139:0/711825103 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7fe2c002fcc0 0x7fe2c0032180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe2c80096a0 tx=0x7fe2c8006b40).stop 2021-11-20T05:58:46.675 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.687+0000 7fe2e3036700 1 -- 172.21.15.139:0/711825103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe2dc0fd2a0 msgr2=0x7fe2dc10a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:46.675 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.687+0000 7fe2e3036700 1 --2- 172.21.15.139:0/711825103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe2dc0fd2a0 0x7fe2dc10a9e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fe2d000b1d0 tx=0x7fe2d0006ee0).stop 2021-11-20T05:58:46.676 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.688+0000 7fe2e3036700 1 -- 172.21.15.139:0/711825103 shutdown_connections 2021-11-20T05:58:46.676 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.688+0000 7fe2e3036700 1 --2- 172.21.15.139:0/711825103 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7fe2c002fcc0 0x7fe2c0032180 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:46.676 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.688+0000 7fe2e3036700 1 --2- 172.21.15.139:0/711825103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe2dc0fd2a0 0x7fe2dc10a9e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:46.676 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.688+0000 7fe2e3036700 1 -- 172.21.15.139:0/711825103 >> 172.21.15.139:0/711825103 conn(0x7fe2dc0f5a80 msgr2=0x7fe2dc0ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:46.677 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.688+0000 7fe2e3036700 1 -- 172.21.15.139:0/711825103 shutdown_connections 2021-11-20T05:58:46.677 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:46.688+0000 7fe2e3036700 1 -- 172.21.15.139:0/711825103 wait complete. 2021-11-20T05:58:46.984 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2021-11-20T05:58:46.996 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:46 smithi139 conmon[33501]: audit 2021-11-20T05:58:46.296692+0000 mon.smithi139 (mon. 2021-11-20T05:58:46.996 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:46 smithi139 conmon[33501]: 0) 24 : audit [INF] from='client.? 172.21.15.139:0/711825103' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2021-11-20T05:58:47.456 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.468+0000 7f66af6f9700 1 Processor -- start 2021-11-20T05:58:47.456 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.468+0000 7f66af6f9700 1 -- start start 2021-11-20T05:58:47.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.468+0000 7f66af6f9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a806fbf0 0x7f66a8100c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:47.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.468+0000 7f66af6f9700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a80fc660 con 0x7f66a806fbf0 2021-11-20T05:58:47.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.469+0000 7f66ad495700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a806fbf0 0x7f66a8100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:47.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.469+0000 7f66ad495700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a806fbf0 0x7f66a8100c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56472/0 (socket says 172.21.15.139:56472) 2021-11-20T05:58:47.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.469+0000 7f66ad495700 1 -- 172.21.15.139:0/3167711653 learned_addr learned my addr 172.21.15.139:0/3167711653 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:47.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.469+0000 7f66ad495700 1 -- 172.21.15.139:0/3167711653 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a8101160 con 0x7f66a806fbf0 2021-11-20T05:58:47.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.469+0000 7f66ad495700 1 --2- 172.21.15.139:0/3167711653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a806fbf0 0x7f66a8100c70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f669800b5f0 tx=0x7f6698006ee0).ready entity=mon.0 client_cookie=d26cbe1a78b9eb7e server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:47.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.469+0000 7f6697fff700 1 -- 172.21.15.139:0/3167711653 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f669800d600 con 0x7f66a806fbf0 2021-11-20T05:58:47.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.469+0000 7f6697fff700 1 -- 172.21.15.139:0/3167711653 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f669800dbd0 con 0x7f66a806fbf0 2021-11-20T05:58:47.459 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.469+0000 7f6697fff700 1 -- 172.21.15.139:0/3167711653 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6698011c30 con 0x7f66a806fbf0 2021-11-20T05:58:47.463 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.470+0000 7f66af6f9700 1 -- 172.21.15.139:0/3167711653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a806fbf0 msgr2=0x7f66a8100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:47.463 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.470+0000 7f66af6f9700 1 --2- 172.21.15.139:0/3167711653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a806fbf0 0x7f66a8100c70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f669800b5f0 tx=0x7f6698006ee0).stop 2021-11-20T05:58:47.463 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.470+0000 7f66af6f9700 1 -- 172.21.15.139:0/3167711653 shutdown_connections 2021-11-20T05:58:47.464 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.470+0000 7f66af6f9700 1 --2- 172.21.15.139:0/3167711653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a806fbf0 0x7f66a8100c70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:47.464 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.470+0000 7f66af6f9700 1 -- 172.21.15.139:0/3167711653 >> 172.21.15.139:0/3167711653 conn(0x7f66a80f6310 msgr2=0x7f66a80f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:47.464 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.471+0000 7f66af6f9700 1 -- 172.21.15.139:0/3167711653 shutdown_connections 2021-11-20T05:58:47.464 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.471+0000 7f66af6f9700 1 -- 172.21.15.139:0/3167711653 wait complete. 2021-11-20T05:58:47.465 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.471+0000 7f66af6f9700 1 Processor -- start 2021-11-20T05:58:47.465 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.471+0000 7f66af6f9700 1 -- start start 2021-11-20T05:58:47.465 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.471+0000 7f66af6f9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a806fbf0 0x7f66a810cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:47.465 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.471+0000 7f66af6f9700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a810d070 con 0x7f66a806fbf0 2021-11-20T05:58:47.465 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.472+0000 7f66ad495700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a806fbf0 0x7f66a810cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:47.466 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.472+0000 7f66ad495700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a806fbf0 0x7f66a810cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56474/0 (socket says 172.21.15.139:56474) 2021-11-20T05:58:47.466 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.472+0000 7f66ad495700 1 -- 172.21.15.139:0/1778985514 learned_addr learned my addr 172.21.15.139:0/1778985514 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:47.466 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.472+0000 7f66ad495700 1 -- 172.21.15.139:0/1778985514 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669800b200 con 0x7f66a806fbf0 2021-11-20T05:58:47.466 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.472+0000 7f66ad495700 1 --2- 172.21.15.139:0/1778985514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a806fbf0 0x7f66a810cb80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f6698011850 tx=0x7f6698006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:47.467 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.472+0000 7f6695ffb700 1 -- 172.21.15.139:0/1778985514 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f669801b570 con 0x7f66a806fbf0 2021-11-20T05:58:47.467 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.472+0000 7f6695ffb700 1 -- 172.21.15.139:0/1778985514 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f669801bb40 con 0x7f66a806fbf0 2021-11-20T05:58:47.467 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.472+0000 7f66af6f9700 1 -- 172.21.15.139:0/1778985514 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a810d270 con 0x7f66a806fbf0 2021-11-20T05:58:47.467 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.472+0000 7f6695ffb700 1 -- 172.21.15.139:0/1778985514 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f669801aa10 con 0x7f66a806fbf0 2021-11-20T05:58:47.467 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.472+0000 7f66af6f9700 1 -- 172.21.15.139:0/1778985514 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a810d680 con 0x7f66a806fbf0 2021-11-20T05:58:47.468 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.473+0000 7f6695ffb700 1 -- 172.21.15.139:0/1778985514 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f669800f070 con 0x7f66a806fbf0 2021-11-20T05:58:47.468 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.473+0000 7f66af6f9700 1 -- 172.21.15.139:0/1778985514 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a8045ad0 con 0x7f66a806fbf0 2021-11-20T05:58:47.468 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.473+0000 7f6695ffb700 1 --2- 172.21.15.139:0/1778985514 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f669002fdb0 0x7f6690032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:47.468 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.474+0000 7f6695ffb700 1 -- 172.21.15.139:0/1778985514 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f669801b960 con 0x7f66a806fbf0 2021-11-20T05:58:47.468 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.474+0000 7f66acc94700 1 -- 172.21.15.139:0/1778985514 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f669002fdb0 msgr2=0x7f6690032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/4049861874 2021-11-20T05:58:47.469 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.474+0000 7f66acc94700 1 --2- 172.21.15.139:0/1778985514 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f669002fdb0 0x7f6690032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T05:58:47.469 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.478+0000 7f6695ffb700 1 -- 172.21.15.139:0/1778985514 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66980107d0 con 0x7f66a806fbf0 2021-11-20T05:58:47.661 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.673+0000 7f66af6f9700 1 -- 172.21.15.139:0/1778985514 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f66a810e790 con 0x7f66a806fbf0 2021-11-20T05:58:47.661 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.674+0000 7f6695ffb700 1 -- 172.21.15.139:0/1778985514 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v4) v1 ==== 56+0+103 (secure 0 0 0) 0x7f6698016070 con 0x7f66a806fbf0 2021-11-20T05:58:47.661 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: { 2021-11-20T05:58:47.662 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 4, 2021-11-20T05:58:47.662 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "available": true, 2021-11-20T05:58:47.663 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "active_name": "smithi139.dplyxn", 2021-11-20T05:58:47.663 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_standby": 0 2021-11-20T05:58:47.663 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: } 2021-11-20T05:58:47.664 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.675+0000 7f66acc94700 1 -- 172.21.15.139:0/1778985514 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f669002fdb0 msgr2=0x7f6690032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/4049861874 2021-11-20T05:58:47.664 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.675+0000 7f66acc94700 1 --2- 172.21.15.139:0/1778985514 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f669002fdb0 0x7f6690032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T05:58:47.665 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.675+0000 7f66af6f9700 1 -- 172.21.15.139:0/1778985514 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f669002fdb0 msgr2=0x7f6690032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T05:58:47.666 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.675+0000 7f66af6f9700 1 --2- 172.21.15.139:0/1778985514 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f669002fdb0 0x7f6690032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:58:47.666 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.675+0000 7f66af6f9700 1 -- 172.21.15.139:0/1778985514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a806fbf0 msgr2=0x7f66a810cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:47.666 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.675+0000 7f66af6f9700 1 --2- 172.21.15.139:0/1778985514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a806fbf0 0x7f66a810cb80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f6698011850 tx=0x7f6698006ee0).stop 2021-11-20T05:58:47.666 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.675+0000 7f66af6f9700 1 -- 172.21.15.139:0/1778985514 shutdown_connections 2021-11-20T05:58:47.667 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.675+0000 7f66af6f9700 1 --2- 172.21.15.139:0/1778985514 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f669002fdb0 0x7f6690032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:58:47.667 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.675+0000 7f66af6f9700 1 --2- 172.21.15.139:0/1778985514 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66a806fbf0 0x7f66a810cb80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:47.667 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.675+0000 7f66af6f9700 1 -- 172.21.15.139:0/1778985514 >> 172.21.15.139:0/1778985514 conn(0x7f66a80f6310 msgr2=0x7f66a80f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:47.667 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.676+0000 7f66af6f9700 1 -- 172.21.15.139:0/1778985514 shutdown_connections 2021-11-20T05:58:47.668 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:47.676+0000 7f66af6f9700 1 -- 172.21.15.139:0/1778985514 wait complete. 2021-11-20T05:58:47.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:47 smithi139 conmon[33501]: audit 2021-11-20T05:58:46.686318+0000 mon.smithi139 (mon.0) 2021-11-20T05:58:47.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:47 smithi139 conmon[33501]: 25 : audit [INF] from='client.? 172.21.15.139:0/711825103' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2021-11-20T05:58:47.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:47 smithi139 conmon[33501]: cluster 2021-11-20T05:58:46.686398+0000 mon.smithi139 (mon.0) 26 : cluster [DBG] mgrmap e4: smithi139.dplyxn(active, since 3s) 2021-11-20T05:58:47.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:47 smithi139 conmon[33501]: audit 2021-11-20T05:58:47.674511+0000 mon.smithi139 (mon.0) 27 : audit [DBG] from='client.? 172.21.15.139:0/1778985514' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-11-20T05:58:48.025 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for the mgr to restart... 2021-11-20T05:58:48.027 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for mgr epoch 4... 2021-11-20T05:58:48.027 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2021-11-20T05:58:48.517 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.528+0000 7f10215ed700 1 Processor -- start 2021-11-20T05:58:48.518 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.528+0000 7f10215ed700 1 -- start start 2021-11-20T05:58:48.518 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.529+0000 7f10215ed700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f101c0fbbd0 0x7f101c0fbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:48.518 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.529+0000 7f10215ed700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c0fc4e0 con 0x7f101c0fbbd0 2021-11-20T05:58:48.518 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.529+0000 7f101affd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f101c0fbbd0 0x7f101c0fbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:48.519 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.529+0000 7f101affd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f101c0fbbd0 0x7f101c0fbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56482/0 (socket says 172.21.15.139:56482) 2021-11-20T05:58:48.519 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.529+0000 7f101affd700 1 -- 172.21.15.139:0/252200991 learned_addr learned my addr 172.21.15.139:0/252200991 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:48.519 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.529+0000 7f101affd700 1 -- 172.21.15.139:0/252200991 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c0fc620 con 0x7f101c0fbbd0 2021-11-20T05:58:48.519 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.530+0000 7f101affd700 1 --2- 172.21.15.139:0/252200991 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f101c0fbbd0 0x7f101c0fbff0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f101000b5f0 tx=0x7f1010006ee0).ready entity=mon.0 client_cookie=e9137bf258126b2a server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:48.519 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.530+0000 7f1019ffb700 1 -- 172.21.15.139:0/252200991 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f101000d600 con 0x7f101c0fbbd0 2021-11-20T05:58:48.520 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.530+0000 7f1019ffb700 1 -- 172.21.15.139:0/252200991 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f101000dbd0 con 0x7f101c0fbbd0 2021-11-20T05:58:48.520 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.530+0000 7f1019ffb700 1 -- 172.21.15.139:0/252200991 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1010011c00 con 0x7f101c0fbbd0 2021-11-20T05:58:48.520 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.530+0000 7f10215ed700 1 -- 172.21.15.139:0/252200991 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f101c0fbbd0 msgr2=0x7f101c0fbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:48.520 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.530+0000 7f10215ed700 1 --2- 172.21.15.139:0/252200991 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f101c0fbbd0 0x7f101c0fbff0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f101000b5f0 tx=0x7f1010006ee0).stop 2021-11-20T05:58:48.521 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.531+0000 7f10215ed700 1 -- 172.21.15.139:0/252200991 shutdown_connections 2021-11-20T05:58:48.521 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.531+0000 7f10215ed700 1 --2- 172.21.15.139:0/252200991 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f101c0fbbd0 0x7f101c0fbff0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:48.521 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.531+0000 7f10215ed700 1 -- 172.21.15.139:0/252200991 >> 172.21.15.139:0/252200991 conn(0x7f101c0f73f0 msgr2=0x7f101c0f9810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:48.521 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.531+0000 7f10215ed700 1 -- 172.21.15.139:0/252200991 shutdown_connections 2021-11-20T05:58:48.521 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.531+0000 7f10215ed700 1 -- 172.21.15.139:0/252200991 wait complete. 2021-11-20T05:58:48.524 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.531+0000 7f10215ed700 1 Processor -- start 2021-11-20T05:58:48.524 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.532+0000 7f10215ed700 1 -- start start 2021-11-20T05:58:48.524 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.532+0000 7f10215ed700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f101c0fbbd0 0x7f101c10a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:48.525 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.532+0000 7f10215ed700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c10ae70 con 0x7f101c0fbbd0 2021-11-20T05:58:48.525 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.532+0000 7f101affd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f101c0fbbd0 0x7f101c10a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:48.525 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.532+0000 7f101affd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f101c0fbbd0 0x7f101c10a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56484/0 (socket says 172.21.15.139:56484) 2021-11-20T05:58:48.525 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.532+0000 7f101affd700 1 -- 172.21.15.139:0/514520958 learned_addr learned my addr 172.21.15.139:0/514520958 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:48.525 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.532+0000 7f101affd700 1 -- 172.21.15.139:0/514520958 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101000b200 con 0x7f101c0fbbd0 2021-11-20T05:58:48.526 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.532+0000 7f101affd700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f101c0fbbd0 0x7f101c10a980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f101000e040 tx=0x7f1010006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:48.526 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.532+0000 7f10037fe700 1 -- 172.21.15.139:0/514520958 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f101001b410 con 0x7f101c0fbbd0 2021-11-20T05:58:48.526 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.532+0000 7f10037fe700 1 -- 172.21.15.139:0/514520958 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f101001b9e0 con 0x7f101c0fbbd0 2021-11-20T05:58:48.526 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.533+0000 7f10037fe700 1 -- 172.21.15.139:0/514520958 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f101001a820 con 0x7f101c0fbbd0 2021-11-20T05:58:48.527 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.533+0000 7f10215ed700 1 -- 172.21.15.139:0/514520958 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101c10b070 con 0x7f101c0fbbd0 2021-11-20T05:58:48.527 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.533+0000 7f10215ed700 1 -- 172.21.15.139:0/514520958 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f101c10b480 con 0x7f101c0fbbd0 2021-11-20T05:58:48.527 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.533+0000 7f10037fe700 1 -- 172.21.15.139:0/514520958 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f101000f070 con 0x7f101c0fbbd0 2021-11-20T05:58:48.527 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.534+0000 7f10037fe700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f10080341c0 0x7f1008036680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:48.527 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.534+0000 7f10037fe700 1 -- 172.21.15.139:0/514520958 --> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1008036d40 con 0x7f10080341c0 2021-11-20T05:58:48.528 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.534+0000 7f10037fe700 1 -- 172.21.15.139:0/514520958 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f10100412b0 con 0x7f101c0fbbd0 2021-11-20T05:58:48.528 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.534+0000 7f101a7fc700 1 -- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f10080341c0 msgr2=0x7f1008036680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/4049861874 2021-11-20T05:58:48.528 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.534+0000 7f101a7fc700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f10080341c0 0x7f1008036680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T05:58:48.721 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.735+0000 7f101a7fc700 1 -- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f10080341c0 msgr2=0x7f1008036680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/4049861874 2021-11-20T05:58:48.722 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:48.735+0000 7f101a7fc700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f10080341c0 0x7f1008036680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T05:58:49.122 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:49.136+0000 7f101a7fc700 1 -- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f10080341c0 msgr2=0x7f1008036680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/4049861874 2021-11-20T05:58:49.123 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:49.136+0000 7f101a7fc700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f10080341c0 0x7f1008036680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T05:58:49.923 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:49.937+0000 7f101a7fc700 1 -- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f10080341c0 msgr2=0x7f1008036680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/4049861874 2021-11-20T05:58:49.924 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:49.937+0000 7f101a7fc700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f10080341c0 0x7f1008036680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-20T05:58:51.525 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:51.538+0000 7f101a7fc700 1 -- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f10080341c0 msgr2=0x7f1008036680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/4049861874 2021-11-20T05:58:51.525 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:51.538+0000 7f101a7fc700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f10080341c0 0x7f1008036680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-20T05:58:51.691 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:51.704+0000 7f10037fe700 1 -- 172.21.15.139:0/514520958 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mgrmap(e 5) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f101001a980 con 0x7f101c0fbbd0 2021-11-20T05:58:51.691 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:51.704+0000 7f10037fe700 1 -- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f10080341c0 msgr2=0x7f1008036680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T05:58:51.692 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:51.704+0000 7f10037fe700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f10080341c0 0x7f1008036680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:58:51.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:51 smithi139 conmon[33501]: cluster 2021-11-20T05:58:51.653371 2021-11-20T05:58:51.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:51 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 28 : cluster [INF] Active manager daemon smithi139.dplyxn restarted 2021-11-20T05:58:51.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:51 smithi139 conmon[33501]: cluster 2021-11-20T 2021-11-20T05:58:51.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:51 smithi139 conmon[33501]: 05:58:51.653761+0000 mon.smithi139 (mon.0) 29 2021-11-20T05:58:51.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:51 smithi139 conmon[33501]: : cluster [INF] Activating manager daemon smithi139.dplyxn 2021-11-20T05:58:51.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:51 smithi139 conmon[33501]: cluster 2021- 2021-11-20T05:58:51.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:51 smithi139 conmon[33501]: 11-20T05:58:51.655487+0000 2021-11-20T05:58:51.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:51 smithi139 conmon[33501]: mon.smithi139 (mon.0) 30 : cluster 2021-11-20T05:58:51.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:51 smithi139 conmon[33501]: [DBG] osdmap e2: 0 total, 0 up, 0 in 2021-11-20T05:58:52.704 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.717+0000 7f10037fe700 1 -- 172.21.15.139:0/514520958 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f1010042b00 con 0x7f101c0fbbd0 2021-11-20T05:58:52.704 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.717+0000 7f10037fe700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f1008037c90 0x7f100803a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:52.704 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.717+0000 7f10037fe700 1 -- 172.21.15.139:0/514520958 --> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1008036d40 con 0x7f1008037c90 2021-11-20T05:58:52.705 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.717+0000 7f101a7fc700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f1008037c90 0x7f100803a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:52.705 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.718+0000 7f101a7fc700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f1008037c90 0x7f100803a080 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f100c006fd0 tx=0x7f100c006d70).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:52.706 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.719+0000 7f10037fe700 1 -- 172.21.15.139:0/514520958 <== mgr.14114 v2:172.21.15.139:6800/1220659339 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f1008036d40 con 0x7f1008037c90 2021-11-20T05:58:52.714 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.725+0000 7f10215ed700 1 -- 172.21.15.139:0/514520958 --> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f101c059100 con 0x7f1008037c90 2021-11-20T05:58:52.715 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.725+0000 7f10037fe700 1 -- 172.21.15.139:0/514520958 <== mgr.14114 v2:172.21.15.139:6800/1220659339 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f101c059100 con 0x7f1008037c90 2021-11-20T05:58:52.715 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: { 2021-11-20T05:58:52.715 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mgrmap_epoch": 6, 2021-11-20T05:58:52.716 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "initialized": true 2021-11-20T05:58:52.716 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: } 2021-11-20T05:58:52.716 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.726+0000 7f10215ed700 1 -- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f1008037c90 msgr2=0x7f100803a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:52.716 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.726+0000 7f10215ed700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f1008037c90 0x7f100803a080 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f100c006fd0 tx=0x7f100c006d70).stop 2021-11-20T05:58:52.716 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.726+0000 7f10215ed700 1 -- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f101c0fbbd0 msgr2=0x7f101c10a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:52.717 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.726+0000 7f10215ed700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f101c0fbbd0 0x7f101c10a980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f101000e040 tx=0x7f1010006ee0).stop 2021-11-20T05:58:52.717 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.726+0000 7f10215ed700 1 -- 172.21.15.139:0/514520958 shutdown_connections 2021-11-20T05:58:52.717 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.726+0000 7f10215ed700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f1008037c90 0x7f100803a080 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:52.718 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.726+0000 7f10215ed700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f101c0fbbd0 0x7f101c10a980 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:52.718 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.726+0000 7f10215ed700 1 --2- 172.21.15.139:0/514520958 >> [v2:172.21.15.139:6800/4049861874,v1:172.21.15.139:6801/4049861874] conn(0x7f10080341c0 0x7f1008036680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:58:52.718 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.726+0000 7f10215ed700 1 -- 172.21.15.139:0/514520958 >> 172.21.15.139:0/514520958 conn(0x7f101c0f73f0 msgr2=0x7f101c0f8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:52.718 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.726+0000 7f10215ed700 1 -- 172.21.15.139:0/514520958 shutdown_connections 2021-11-20T05:58:52.718 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:52.726+0000 7f10215ed700 1 -- 172.21.15.139:0/514520958 wait complete. 2021-11-20T05:58:53.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: cluster 2021-11-20T05:58:51.706409+0000 mon.smithi139 (mon.0) 31 : cluster [DBG] mgrmap e5: smithi139.dplyxn(active, starting, since 0.0527234s) 2021-11-20T05:58:53.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: audit 2021-11-20T05:58:51.707954+0000 mon.smithi139 (mon.0) 32 : audit [DBG] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi139"}]: dispatch 2021-11-20T05:58:53.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: audit 2021-11-20T05:58:51.708628+0000 mon.smithi139 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mgr metadata", "who": "smithi139.dplyxn", "id": "smithi139.dplyxn"}]: dispatch 2021-11-20T05:58:53.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: audit 2021-11-20T05:58:51.709240+0000 2021-11-20T05:58:53.030 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: mon.smithi139 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T05:58:53.030 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: audit 2021-11-20T05:58:51 2021-11-20T05:58:53.030 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: .709373+0000 mon.smithi139 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T05:58:53.030 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T05:58:53.031 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: 20T05:58:51.709503+0000 mon.smithi139 (mon.0) 36 : audit 2021-11-20T05:58:53.031 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: [DBG] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T05:58:53.031 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: cluster 2021- 2021-11-20T05:58:53.031 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: 11-20T05:58:51.722931+0000 mon.smithi139 (mon.0) 37 2021-11-20T05:58:53.031 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: : cluster [INF] Manager daemon smithi139.dplyxn is now available 2021-11-20T05:58:53.032 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: audit 2021-11-20T05:58:53.032 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: 2021-11-20T05:58:51.739771+0000 mon.smithi139 (mon. 2021-11-20T05:58:53.032 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: 0) 38 : audit [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' 2021-11-20T05:58:53.032 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: audit 2021-11-20T 2021-11-20T05:58:53.032 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: 05:58:51.745182+0000 mon.smithi139 (mon.0 2021-11-20T05:58:53.033 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: ) 39 : audit [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' 2021-11-20T05:58:53.033 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: audit 2021-11-20T05:58:51 2021-11-20T05:58:53.033 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: .750454+0000 mon.smithi139 (mon.0) 40 : audit 2021-11-20T05:58:53.034 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' 2021-11-20T05:58:53.034 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: audit 2021- 2021-11-20T05:58:53.034 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: 11-20T05:58:51.841221+0000 mon.smithi139 (mon.0) 41 : 2021-11-20T05:58:53.034 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: audit [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' 2021-11-20T05:58:53.035 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: audit 2021-11-20T05:58:53.035 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: 2021-11-20T05:58:51.841880+0000 mon.smithi139 (mon. 2021-11-20T05:58:53.035 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: 0) 42 : audit [DBG] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:58:53.035 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: audit 2021-11-20T05:58:51 2021-11-20T05:58:53.036 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: .842940+0000 mon.smithi139 (mon.0) 43 : audit 2021-11-20T05:58:53.036 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: [DBG] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:58:53.036 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: audit 2021-11-20T05:58:53.036 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: 2021-11-20T05:58:51.844448+0000 mon.smithi139 (mon.0 2021-11-20T05:58:53.037 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: ) 44 : audit [DBG] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:58:53.037 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: audit 2021-11-20T05:58:51.854347+0000 mon.smithi139 (mon. 2021-11-20T05:58:53.037 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: 0) 45 : audit [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:58:53.037 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: audit 2021-11-20T05:58:51 2021-11-20T05:58:53.037 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: .855512+0000 mon.smithi139 (mon.0) 46 : audit 2021-11-20T05:58:53.038 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:52 smithi139 conmon[33501]: [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T05:58:54.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:53 smithi139 conmon[33501]: audit 2021-11-20T05:58:52.719527+0000 mgr.smithi139.dplyxn (mgr.14114) 1 : audit 2021-11-20T05:58:54.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:53 smithi139 conmon[33501]: [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-11-20T05:58:54.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:53 smithi139 conmon[33501]: cluster 2021-11-20T05:58:52.720334+0000 mon.smithi139 (mon.0) 47 : cluster [DBG] mgrmap e6: smithi139.dplyxn(active, since 1.06655s) 2021-11-20T05:58:54.576 INFO:teuthology.orchestra.run.smithi139.stderr:mgr epoch 4 is available 2021-11-20T05:58:54.576 INFO:teuthology.orchestra.run.smithi139.stderr:Setting orchestrator backend to cephadm... 2021-11-20T05:58:54.577 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2021-11-20T05:58:55.026 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:54 smithi139 conmon[33501]: audit 2021-11-20T05:58:52.726439+0000 mgr.smithi139.dplyxn (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-11-20T05:58:55.026 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:54 smithi139 conmon[33501]: cluster 2021-11-20T05:58:53.724672+0000 mon.smithi139 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi139.dplyxn(active, since 2s) 2021-11-20T05:58:56.592 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.519+0000 7fbe207ea700 1 Processor -- start 2021-11-20T05:58:56.593 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.519+0000 7fbe207ea700 1 -- start start 2021-11-20T05:58:56.593 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.519+0000 7fbe207ea700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe1806fc30 0x7fbe180fe830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:56.593 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.519+0000 7fbe207ea700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe180fed20 con 0x7fbe1806fc30 2021-11-20T05:58:56.594 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.520+0000 7fbe1e586700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe1806fc30 0x7fbe180fe830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:56.594 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.520+0000 7fbe1e586700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe1806fc30 0x7fbe180fe830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56520/0 (socket says 172.21.15.139:56520) 2021-11-20T05:58:56.594 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.520+0000 7fbe1e586700 1 -- 172.21.15.139:0/2950263202 learned_addr learned my addr 172.21.15.139:0/2950263202 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:56.594 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.520+0000 7fbe1e586700 1 -- 172.21.15.139:0/2950263202 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe180fee60 con 0x7fbe1806fc30 2021-11-20T05:58:56.595 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.520+0000 7fbe1e586700 1 --2- 172.21.15.139:0/2950263202 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe1806fc30 0x7fbe180fe830 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fbe14009960 tx=0x7fbe140092d0).ready entity=mon.0 client_cookie=5cc6cac50b3ad1b5 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:56.595 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.520+0000 7fbe1d584700 1 -- 172.21.15.139:0/2950263202 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe1400b9c0 con 0x7fbe1806fc30 2021-11-20T05:58:56.595 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.520+0000 7fbe1d584700 1 -- 172.21.15.139:0/2950263202 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbe14011400 con 0x7fbe1806fc30 2021-11-20T05:58:56.595 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.520+0000 7fbe1d584700 1 -- 172.21.15.139:0/2950263202 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe140103a0 con 0x7fbe1806fc30 2021-11-20T05:58:56.596 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.521+0000 7fbe207ea700 1 -- 172.21.15.139:0/2950263202 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe1806fc30 msgr2=0x7fbe180fe830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:56.596 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.521+0000 7fbe207ea700 1 --2- 172.21.15.139:0/2950263202 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe1806fc30 0x7fbe180fe830 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fbe14009960 tx=0x7fbe140092d0).stop 2021-11-20T05:58:56.596 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.521+0000 7fbe207ea700 1 -- 172.21.15.139:0/2950263202 shutdown_connections 2021-11-20T05:58:56.596 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.521+0000 7fbe207ea700 1 --2- 172.21.15.139:0/2950263202 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe1806fc30 0x7fbe180fe830 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:56.596 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.521+0000 7fbe207ea700 1 -- 172.21.15.139:0/2950263202 >> 172.21.15.139:0/2950263202 conn(0x7fbe180f6310 msgr2=0x7fbe180f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:56.597 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.521+0000 7fbe207ea700 1 -- 172.21.15.139:0/2950263202 shutdown_connections 2021-11-20T05:58:56.597 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.521+0000 7fbe207ea700 1 -- 172.21.15.139:0/2950263202 wait complete. 2021-11-20T05:58:56.597 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.521+0000 7fbe207ea700 1 Processor -- start 2021-11-20T05:58:56.597 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.521+0000 7fbe207ea700 1 -- start start 2021-11-20T05:58:56.598 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.521+0000 7fbe207ea700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe1806fc30 0x7fbe1810a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:56.598 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.522+0000 7fbe207ea700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1810ae50 con 0x7fbe1806fc30 2021-11-20T05:58:56.598 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.522+0000 7fbe1e586700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe1806fc30 0x7fbe1810a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:56.598 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.522+0000 7fbe1e586700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe1806fc30 0x7fbe1810a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56522/0 (socket says 172.21.15.139:56522) 2021-11-20T05:58:56.598 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.522+0000 7fbe1e586700 1 -- 172.21.15.139:0/87068507 learned_addr learned my addr 172.21.15.139:0/87068507 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:56.599 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.522+0000 7fbe1e586700 1 -- 172.21.15.139:0/87068507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe14009570 con 0x7fbe1806fc30 2021-11-20T05:58:56.599 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.522+0000 7fbe1e586700 1 --2- 172.21.15.139:0/87068507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe1806fc30 0x7fbe1810a960 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fbe14000f80 tx=0x7fbe140092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:56.599 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.522+0000 7fbe06ffd700 1 -- 172.21.15.139:0/87068507 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe14010820 con 0x7fbe1806fc30 2021-11-20T05:58:56.600 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.522+0000 7fbe06ffd700 1 -- 172.21.15.139:0/87068507 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbe14010df0 con 0x7fbe1806fc30 2021-11-20T05:58:56.600 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.522+0000 7fbe06ffd700 1 -- 172.21.15.139:0/87068507 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe14023410 con 0x7fbe1806fc30 2021-11-20T05:58:56.600 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.522+0000 7fbe207ea700 1 -- 172.21.15.139:0/87068507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1810b050 con 0x7fbe1806fc30 2021-11-20T05:58:56.600 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.522+0000 7fbe207ea700 1 -- 172.21.15.139:0/87068507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe1810b460 con 0x7fbe1806fc30 2021-11-20T05:58:56.601 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.523+0000 7fbe06ffd700 1 -- 172.21.15.139:0/87068507 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fbe1400f070 con 0x7fbe1806fc30 2021-11-20T05:58:56.601 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.523+0000 7fbe06ffd700 1 --2- 172.21.15.139:0/87068507 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fbe0002fdb0 0x7fbe00032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:56.601 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.523+0000 7fbe06ffd700 1 -- 172.21.15.139:0/87068507 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fbe14041e60 con 0x7fbe1806fc30 2021-11-20T05:58:56.601 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.523+0000 7fbe207ea700 1 -- 172.21.15.139:0/87068507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe18045ad0 con 0x7fbe1806fc30 2021-11-20T05:58:56.602 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.523+0000 7fbe1dd85700 1 --2- 172.21.15.139:0/87068507 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fbe0002fdb0 0x7fbe00032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:56.602 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.524+0000 7fbe1dd85700 1 --2- 172.21.15.139:0/87068507 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fbe0002fdb0 0x7fbe00032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbe080096a0 tx=0x7fbe08006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:56.602 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.527+0000 7fbe06ffd700 1 -- 172.21.15.139:0/87068507 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe1401e020 con 0x7fbe1806fc30 2021-11-20T05:58:56.695 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.709+0000 7fbe207ea700 1 -- 172.21.15.139:0/87068507 --> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe181047d0 con 0x7fbe0002fdb0 2021-11-20T05:58:56.700 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.714+0000 7fbe06ffd700 1 -- 172.21.15.139:0/87068507 <== mgr.14114 v2:172.21.15.139:6800/1220659339 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbe181047d0 con 0x7fbe0002fdb0 2021-11-20T05:58:56.703 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.716+0000 7fbe207ea700 1 -- 172.21.15.139:0/87068507 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fbe0002fdb0 msgr2=0x7fbe00032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:56.703 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.716+0000 7fbe207ea700 1 --2- 172.21.15.139:0/87068507 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fbe0002fdb0 0x7fbe00032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbe080096a0 tx=0x7fbe08006b40).stop 2021-11-20T05:58:56.704 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.716+0000 7fbe207ea700 1 -- 172.21.15.139:0/87068507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe1806fc30 msgr2=0x7fbe1810a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:56.704 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.716+0000 7fbe207ea700 1 --2- 172.21.15.139:0/87068507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe1806fc30 0x7fbe1810a960 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fbe14000f80 tx=0x7fbe140092d0).stop 2021-11-20T05:58:56.704 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.716+0000 7fbe207ea700 1 -- 172.21.15.139:0/87068507 shutdown_connections 2021-11-20T05:58:56.704 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.716+0000 7fbe207ea700 1 --2- 172.21.15.139:0/87068507 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fbe0002fdb0 0x7fbe00032270 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:56.705 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.716+0000 7fbe207ea700 1 --2- 172.21.15.139:0/87068507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe1806fc30 0x7fbe1810a960 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:56.705 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.716+0000 7fbe207ea700 1 -- 172.21.15.139:0/87068507 >> 172.21.15.139:0/87068507 conn(0x7fbe180f6310 msgr2=0x7fbe180f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:56.705 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.716+0000 7fbe207ea700 1 -- 172.21.15.139:0/87068507 shutdown_connections 2021-11-20T05:58:56.705 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:56.716+0000 7fbe207ea700 1 -- 172.21.15.139:0/87068507 wait complete. 2021-11-20T05:58:58.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:57 smithi139 conmon[33501]: audit 2021-11-20T05:58:56.710477+0000 mgr.smithi139.dplyxn (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:58:58.230 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:57 smithi139 conmon[33501]: 2021-11-20T05:58:58.231 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:57 smithi139 conmon[33501]: audit 2021-11-20T05:58:56.714358+0000 mon.smithi139 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' 2021-11-20T05:58:58.231 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:57 smithi139 conmon[33501]: audit 2021- 2021-11-20T05:58:58.231 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:58:57 smithi139 conmon[33501]: 11-20T05:58:56.718007+0000 mon.smithi139 (mon.0) 50 : audit [DBG] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:58:58.426 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2021-11-20T05:58:59.902 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.911+0000 7f7438a6f700 1 Processor -- start 2021-11-20T05:58:59.902 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.911+0000 7f7438a6f700 1 -- start start 2021-11-20T05:58:59.902 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.911+0000 7f7438a6f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f743406fbf0 0x7f74340fe640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:59.903 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.911+0000 7f7438a6f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74340feb30 con 0x7f743406fbf0 2021-11-20T05:58:59.903 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.911+0000 7f74337fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f743406fbf0 0x7f74340fe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:59.903 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.911+0000 7f74337fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f743406fbf0 0x7f74340fe640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56526/0 (socket says 172.21.15.139:56526) 2021-11-20T05:58:59.903 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.911+0000 7f74337fe700 1 -- 172.21.15.139:0/2832084485 learned_addr learned my addr 172.21.15.139:0/2832084485 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:59.904 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.911+0000 7f74337fe700 1 -- 172.21.15.139:0/2832084485 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74340fec70 con 0x7f743406fbf0 2021-11-20T05:58:59.904 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.912+0000 7f74337fe700 1 --2- 172.21.15.139:0/2832084485 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f743406fbf0 0x7f74340fe640 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f742400b5f0 tx=0x7f7424006ee0).ready entity=mon.0 client_cookie=ae87e54d7da1b11e server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:59.904 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.912+0000 7f74327fc700 1 -- 172.21.15.139:0/2832084485 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f742400d600 con 0x7f743406fbf0 2021-11-20T05:58:59.904 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.912+0000 7f74327fc700 1 -- 172.21.15.139:0/2832084485 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f742400dbd0 con 0x7f743406fbf0 2021-11-20T05:58:59.904 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.912+0000 7f74327fc700 1 -- 172.21.15.139:0/2832084485 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7424011c30 con 0x7f743406fbf0 2021-11-20T05:58:59.905 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.912+0000 7f7438a6f700 1 -- 172.21.15.139:0/2832084485 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f743406fbf0 msgr2=0x7f74340fe640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:58:59.905 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.912+0000 7f7438a6f700 1 --2- 172.21.15.139:0/2832084485 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f743406fbf0 0x7f74340fe640 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f742400b5f0 tx=0x7f7424006ee0).stop 2021-11-20T05:58:59.905 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.912+0000 7f7438a6f700 1 -- 172.21.15.139:0/2832084485 shutdown_connections 2021-11-20T05:58:59.905 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.912+0000 7f7438a6f700 1 --2- 172.21.15.139:0/2832084485 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f743406fbf0 0x7f74340fe640 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:58:59.906 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.912+0000 7f7438a6f700 1 -- 172.21.15.139:0/2832084485 >> 172.21.15.139:0/2832084485 conn(0x7f74340f60e0 msgr2=0x7f74340f8500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:58:59.906 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.912+0000 7f7438a6f700 1 -- 172.21.15.139:0/2832084485 shutdown_connections 2021-11-20T05:58:59.906 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.912+0000 7f7438a6f700 1 -- 172.21.15.139:0/2832084485 wait complete. 2021-11-20T05:58:59.906 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.913+0000 7f7438a6f700 1 Processor -- start 2021-11-20T05:58:59.906 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.913+0000 7f7438a6f700 1 -- start start 2021-11-20T05:58:59.907 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.913+0000 7f7438a6f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f743406fbf0 0x7f743410a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:59.907 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.913+0000 7f7438a6f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743410ad40 con 0x7f743406fbf0 2021-11-20T05:58:59.907 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.913+0000 7f74337fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f743406fbf0 0x7f743410a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:59.907 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.913+0000 7f74337fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f743406fbf0 0x7f743410a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56528/0 (socket says 172.21.15.139:56528) 2021-11-20T05:58:59.908 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.913+0000 7f74337fe700 1 -- 172.21.15.139:0/1036681615 learned_addr learned my addr 172.21.15.139:0/1036681615 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:58:59.908 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.914+0000 7f74337fe700 1 -- 172.21.15.139:0/1036681615 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742400b200 con 0x7f743406fbf0 2021-11-20T05:58:59.908 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.914+0000 7f74337fe700 1 --2- 172.21.15.139:0/1036681615 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f743406fbf0 0x7f743410a850 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f7424011550 tx=0x7f7424006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:59.908 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.914+0000 7f741bfff700 1 -- 172.21.15.139:0/1036681615 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74240116c0 con 0x7f743406fbf0 2021-11-20T05:58:59.908 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.914+0000 7f741bfff700 1 -- 172.21.15.139:0/1036681615 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f742400d9f0 con 0x7f743406fbf0 2021-11-20T05:58:59.909 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.914+0000 7f7438a6f700 1 -- 172.21.15.139:0/1036681615 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f743410af40 con 0x7f743406fbf0 2021-11-20T05:58:59.909 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.914+0000 7f741bfff700 1 -- 172.21.15.139:0/1036681615 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f742401b780 con 0x7f743406fbf0 2021-11-20T05:58:59.909 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.914+0000 7f7438a6f700 1 -- 172.21.15.139:0/1036681615 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f743410b350 con 0x7f743406fbf0 2021-11-20T05:58:59.909 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.915+0000 7f741bfff700 1 -- 172.21.15.139:0/1036681615 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f742400f070 con 0x7f743406fbf0 2021-11-20T05:58:59.910 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.915+0000 7f741bfff700 1 --2- 172.21.15.139:0/1036681615 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f742002fd50 0x7f7420032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:58:59.910 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.915+0000 7f7438a6f700 1 -- 172.21.15.139:0/1036681615 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7434045ad0 con 0x7f743406fbf0 2021-11-20T05:58:59.911 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.915+0000 7f741bfff700 1 -- 172.21.15.139:0/1036681615 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f7424041290 con 0x7f743406fbf0 2021-11-20T05:58:59.911 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.915+0000 7f7432ffd700 1 --2- 172.21.15.139:0/1036681615 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f742002fd50 0x7f7420032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:58:59.911 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.915+0000 7f7432ffd700 1 --2- 172.21.15.139:0/1036681615 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f742002fd50 0x7f7420032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f74280096a0 tx=0x7f7428006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:58:59.911 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:58:59.919+0000 7f741bfff700 1 -- 172.21.15.139:0/1036681615 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74240228c0 con 0x7f743406fbf0 2021-11-20T05:59:00.056 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.068+0000 7f7438a6f700 1 -- 172.21.15.139:0/1036681615 --> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f743410bc30 con 0x7f742002fd50 2021-11-20T05:59:00.056 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.069+0000 7f741bfff700 1 -- 172.21.15.139:0/1036681615 <== mgr.14114 v2:172.21.15.139:6800/1220659339 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f743410bc30 con 0x7f742002fd50 2021-11-20T05:59:00.057 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: value unchanged 2021-11-20T05:59:00.058 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.071+0000 7f7438a6f700 1 -- 172.21.15.139:0/1036681615 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f742002fd50 msgr2=0x7f7420032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:00.059 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.071+0000 7f7438a6f700 1 --2- 172.21.15.139:0/1036681615 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f742002fd50 0x7f7420032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f74280096a0 tx=0x7f7428006b40).stop 2021-11-20T05:59:00.059 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.071+0000 7f7438a6f700 1 -- 172.21.15.139:0/1036681615 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f743406fbf0 msgr2=0x7f743410a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:00.059 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.071+0000 7f7438a6f700 1 --2- 172.21.15.139:0/1036681615 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f743406fbf0 0x7f743410a850 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f7424011550 tx=0x7f7424006ee0).stop 2021-11-20T05:59:00.060 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.071+0000 7f7438a6f700 1 -- 172.21.15.139:0/1036681615 shutdown_connections 2021-11-20T05:59:00.060 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.071+0000 7f7438a6f700 1 --2- 172.21.15.139:0/1036681615 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f742002fd50 0x7f7420032210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:00.060 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.071+0000 7f7438a6f700 1 --2- 172.21.15.139:0/1036681615 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f743406fbf0 0x7f743410a850 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:00.060 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.071+0000 7f7438a6f700 1 -- 172.21.15.139:0/1036681615 >> 172.21.15.139:0/1036681615 conn(0x7f74340f60e0 msgr2=0x7f74340f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:00.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.071+0000 7f7438a6f700 1 -- 172.21.15.139:0/1036681615 shutdown_connections 2021-11-20T05:59:00.061 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.071+0000 7f7438a6f700 1 -- 172.21.15.139:0/1036681615 wait complete. 2021-11-20T05:59:00.467 INFO:teuthology.orchestra.run.smithi139.stderr:Generating ssh key... 2021-11-20T05:59:00.468 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2021-11-20T05:59:00.893 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:00 smithi139 conmon[33501]: audit 2021-11-20T05:59:00.069956+0000 mgr.smithi139.dplyxn (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:59:00.897 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.906+0000 7f96f5061700 1 Processor -- start 2021-11-20T05:59:00.897 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.906+0000 7f96f5061700 1 -- start start 2021-11-20T05:59:00.898 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.906+0000 7f96f5061700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96f0071cd0 0x7f96f0070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:00.898 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.906+0000 7f96f5061700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f00720f0 con 0x7f96f0071cd0 2021-11-20T05:59:00.898 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.907+0000 7f96eed9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96f0071cd0 0x7f96f0070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:00.898 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.907+0000 7f96eed9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96f0071cd0 0x7f96f0070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56532/0 (socket says 172.21.15.139:56532) 2021-11-20T05:59:00.899 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.907+0000 7f96eed9d700 1 -- 172.21.15.139:0/2083948290 learned_addr learned my addr 172.21.15.139:0/2083948290 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:00.899 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.907+0000 7f96eed9d700 1 -- 172.21.15.139:0/2083948290 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f0070760 con 0x7f96f0071cd0 2021-11-20T05:59:00.899 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.907+0000 7f96eed9d700 1 --2- 172.21.15.139:0/2083948290 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96f0071cd0 0x7f96f0070270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f96e000b5f0 tx=0x7f96e0006ee0).ready entity=mon.0 client_cookie=76a52fdd533ff079 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:00.899 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.907+0000 7f96edd9b700 1 -- 172.21.15.139:0/2083948290 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96e000d600 con 0x7f96f0071cd0 2021-11-20T05:59:00.900 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.907+0000 7f96edd9b700 1 -- 172.21.15.139:0/2083948290 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f96e000dbd0 con 0x7f96f0071cd0 2021-11-20T05:59:00.900 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.907+0000 7f96edd9b700 1 -- 172.21.15.139:0/2083948290 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96e0011c00 con 0x7f96f0071cd0 2021-11-20T05:59:00.900 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.908+0000 7f96f5061700 1 -- 172.21.15.139:0/2083948290 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96f0071cd0 msgr2=0x7f96f0070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:00.900 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.908+0000 7f96f5061700 1 --2- 172.21.15.139:0/2083948290 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96f0071cd0 0x7f96f0070270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f96e000b5f0 tx=0x7f96e0006ee0).stop 2021-11-20T05:59:00.900 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.908+0000 7f96f5061700 1 -- 172.21.15.139:0/2083948290 shutdown_connections 2021-11-20T05:59:00.901 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.908+0000 7f96f5061700 1 --2- 172.21.15.139:0/2083948290 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96f0071cd0 0x7f96f0070270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:00.901 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.908+0000 7f96f5061700 1 -- 172.21.15.139:0/2083948290 >> 172.21.15.139:0/2083948290 conn(0x7f96f00f73e0 msgr2=0x7f96f00f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:00.901 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.908+0000 7f96f5061700 1 -- 172.21.15.139:0/2083948290 shutdown_connections 2021-11-20T05:59:00.902 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.908+0000 7f96f5061700 1 -- 172.21.15.139:0/2083948290 wait complete. 2021-11-20T05:59:00.902 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.909+0000 7f96f5061700 1 Processor -- start 2021-11-20T05:59:00.902 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.909+0000 7f96f5061700 1 -- start start 2021-11-20T05:59:00.902 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.909+0000 7f96f5061700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96f010cd10 0x7f96f010d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:00.903 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.909+0000 7f96f5061700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f010d620 con 0x7f96f010cd10 2021-11-20T05:59:00.903 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.909+0000 7f96eed9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96f010cd10 0x7f96f010d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:00.903 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.909+0000 7f96eed9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96f010cd10 0x7f96f010d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56534/0 (socket says 172.21.15.139:56534) 2021-11-20T05:59:00.903 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.909+0000 7f96eed9d700 1 -- 172.21.15.139:0/1175071533 learned_addr learned my addr 172.21.15.139:0/1175071533 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:00.904 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.910+0000 7f96eed9d700 1 -- 172.21.15.139:0/1175071533 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e000b200 con 0x7f96f010cd10 2021-11-20T05:59:00.904 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.910+0000 7f96eed9d700 1 --2- 172.21.15.139:0/1175071533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96f010cd10 0x7f96f010d130 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f96e0011820 tx=0x7f96e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:00.904 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.910+0000 7f96e77fe700 1 -- 172.21.15.139:0/1175071533 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96e000d9f0 con 0x7f96f010cd10 2021-11-20T05:59:00.905 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.910+0000 7f96e77fe700 1 -- 172.21.15.139:0/1175071533 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f96e001a9f0 con 0x7f96f010cd10 2021-11-20T05:59:00.905 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.910+0000 7f96f5061700 1 -- 172.21.15.139:0/1175071533 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f010d820 con 0x7f96f010cd10 2021-11-20T05:59:00.905 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.910+0000 7f96e77fe700 1 -- 172.21.15.139:0/1175071533 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96e001b7b0 con 0x7f96f010cd10 2021-11-20T05:59:00.905 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.910+0000 7f96f5061700 1 -- 172.21.15.139:0/1175071533 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f010e1b0 con 0x7f96f010cd10 2021-11-20T05:59:00.905 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.911+0000 7f96e77fe700 1 -- 172.21.15.139:0/1175071533 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f96e000f070 con 0x7f96f010cd10 2021-11-20T05:59:00.906 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.911+0000 7f96e77fe700 1 --2- 172.21.15.139:0/1175071533 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f96d002fd50 0x7f96d0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:00.906 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.911+0000 7f96f5061700 1 -- 172.21.15.139:0/1175071533 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f0045ad0 con 0x7f96f010cd10 2021-11-20T05:59:00.906 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.911+0000 7f96e77fe700 1 -- 172.21.15.139:0/1175071533 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f96e00412f0 con 0x7f96f010cd10 2021-11-20T05:59:00.907 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.912+0000 7f96ee59c700 1 --2- 172.21.15.139:0/1175071533 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f96d002fd50 0x7f96d0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:00.907 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.912+0000 7f96ee59c700 1 --2- 172.21.15.139:0/1175071533 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f96d002fd50 0x7f96d0032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f96d80096a0 tx=0x7f96d8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:00.907 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:00.915+0000 7f96e77fe700 1 -- 172.21.15.139:0/1175071533 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96e0016080 con 0x7f96f010cd10 2021-11-20T05:59:01.052 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:01.065+0000 7f96f5061700 1 -- 172.21.15.139:0/1175071533 --> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f96f0106a30 con 0x7f96d002fd50 2021-11-20T05:59:01.404 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:01.417+0000 7f96e77fe700 1 -- 172.21.15.139:0/1175071533 <== mgr.14114 v2:172.21.15.139:6800/1220659339 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f96f0106a30 con 0x7f96d002fd50 2021-11-20T05:59:01.406 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:01.419+0000 7f96f5061700 1 -- 172.21.15.139:0/1175071533 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f96d002fd50 msgr2=0x7f96d0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:01.406 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:01.419+0000 7f96f5061700 1 --2- 172.21.15.139:0/1175071533 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f96d002fd50 0x7f96d0032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f96d80096a0 tx=0x7f96d8006b40).stop 2021-11-20T05:59:01.407 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:01.419+0000 7f96f5061700 1 -- 172.21.15.139:0/1175071533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96f010cd10 msgr2=0x7f96f010d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:01.407 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:01.419+0000 7f96f5061700 1 --2- 172.21.15.139:0/1175071533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96f010cd10 0x7f96f010d130 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f96e0011820 tx=0x7f96e0006ee0).stop 2021-11-20T05:59:01.407 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:01.419+0000 7f96f5061700 1 -- 172.21.15.139:0/1175071533 shutdown_connections 2021-11-20T05:59:01.407 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:01.419+0000 7f96f5061700 1 --2- 172.21.15.139:0/1175071533 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f96d002fd50 0x7f96d0032210 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:01.408 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:01.419+0000 7f96f5061700 1 --2- 172.21.15.139:0/1175071533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96f010cd10 0x7f96f010d130 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:01.408 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:01.419+0000 7f96f5061700 1 -- 172.21.15.139:0/1175071533 >> 172.21.15.139:0/1175071533 conn(0x7f96f00f73e0 msgr2=0x7f96f00f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:01.408 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:01.419+0000 7f96f5061700 1 -- 172.21.15.139:0/1175071533 shutdown_connections 2021-11-20T05:59:01.409 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:01.419+0000 7f96f5061700 1 -- 172.21.15.139:0/1175071533 wait complete. 2021-11-20T05:59:01.705 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2021-11-20T05:59:02.124 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.132+0000 7fe3e1b39700 1 Processor -- start 2021-11-20T05:59:02.126 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.132+0000 7fe3e1b39700 1 -- start start 2021-11-20T05:59:02.127 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.132+0000 7fe3e1b39700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3dc06fcd0 0x7fe3dc0fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:02.127 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.132+0000 7fe3e1b39700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc0fed20 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.127 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.133+0000 7fe3db7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3dc06fcd0 0x7fe3dc0fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:02.127 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.133+0000 7fe3db7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3dc06fcd0 0x7fe3dc0fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56538/0 (socket says 172.21.15.139:56538) 2021-11-20T05:59:02.127 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.133+0000 7fe3db7fe700 1 -- 172.21.15.139:0/3496775169 learned_addr learned my addr 172.21.15.139:0/3496775169 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:02.129 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.133+0000 7fe3db7fe700 1 -- 172.21.15.139:0/3496775169 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc0fee60 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.129 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.133+0000 7fe3db7fe700 1 --2- 172.21.15.139:0/3496775169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3dc06fcd0 0x7fe3dc0fe7a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fe3cc00b5f0 tx=0x7fe3cc006ee0).ready entity=mon.0 client_cookie=7e92e279c05baf58 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:02.129 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.133+0000 7fe3daffd700 1 -- 172.21.15.139:0/3496775169 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3cc00d600 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.130 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.133+0000 7fe3daffd700 1 -- 172.21.15.139:0/3496775169 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe3cc00dbd0 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.130 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.133+0000 7fe3daffd700 1 -- 172.21.15.139:0/3496775169 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3cc011c00 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.131 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.134+0000 7fe3e1b39700 1 -- 172.21.15.139:0/3496775169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3dc06fcd0 msgr2=0x7fe3dc0fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:02.131 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.134+0000 7fe3e1b39700 1 --2- 172.21.15.139:0/3496775169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3dc06fcd0 0x7fe3dc0fe7a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fe3cc00b5f0 tx=0x7fe3cc006ee0).stop 2021-11-20T05:59:02.132 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.134+0000 7fe3e1b39700 1 -- 172.21.15.139:0/3496775169 shutdown_connections 2021-11-20T05:59:02.132 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.134+0000 7fe3e1b39700 1 --2- 172.21.15.139:0/3496775169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3dc06fcd0 0x7fe3dc0fe7a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:02.132 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.134+0000 7fe3e1b39700 1 -- 172.21.15.139:0/3496775169 >> 172.21.15.139:0/3496775169 conn(0x7fe3dc0f6280 msgr2=0x7fe3dc0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:02.133 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.134+0000 7fe3e1b39700 1 -- 172.21.15.139:0/3496775169 shutdown_connections 2021-11-20T05:59:02.133 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.134+0000 7fe3e1b39700 1 -- 172.21.15.139:0/3496775169 wait complete. 2021-11-20T05:59:02.134 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.134+0000 7fe3e1b39700 1 Processor -- start 2021-11-20T05:59:02.134 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.134+0000 7fe3e1b39700 1 -- start start 2021-11-20T05:59:02.135 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.135+0000 7fe3e1b39700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3dc06fcd0 0x7fe3dc10a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:02.135 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.135+0000 7fe3e1b39700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc10ae90 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.136 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.135+0000 7fe3db7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3dc06fcd0 0x7fe3dc10a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:02.136 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.135+0000 7fe3db7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3dc06fcd0 0x7fe3dc10a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56540/0 (socket says 172.21.15.139:56540) 2021-11-20T05:59:02.137 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.135+0000 7fe3db7fe700 1 -- 172.21.15.139:0/4041042253 learned_addr learned my addr 172.21.15.139:0/4041042253 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:02.137 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.135+0000 7fe3db7fe700 1 -- 172.21.15.139:0/4041042253 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc00b200 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.137 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.135+0000 7fe3db7fe700 1 --2- 172.21.15.139:0/4041042253 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3dc06fcd0 0x7fe3dc10a9a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fe3cc00e040 tx=0x7fe3cc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:02.138 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.135+0000 7fe3d8ff9700 1 -- 172.21.15.139:0/4041042253 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3cc01b410 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.138 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.135+0000 7fe3d8ff9700 1 -- 172.21.15.139:0/4041042253 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe3cc01b9e0 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.138 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.135+0000 7fe3d8ff9700 1 -- 172.21.15.139:0/4041042253 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3cc01a770 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.138 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.135+0000 7fe3e1b39700 1 -- 172.21.15.139:0/4041042253 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3dc10b090 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.139 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.135+0000 7fe3e1b39700 1 -- 172.21.15.139:0/4041042253 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3dc10b4a0 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.139 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.136+0000 7fe3d8ff9700 1 -- 172.21.15.139:0/4041042253 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe3cc00f070 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.139 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.136+0000 7fe3d8ff9700 1 --2- 172.21.15.139:0/4041042253 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fe3bc02fdb0 0x7fe3bc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:02.139 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.136+0000 7fe3d8ff9700 1 -- 172.21.15.139:0/4041042253 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe3cc041000 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.140 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.136+0000 7fe3d2ffd700 1 --2- 172.21.15.139:0/4041042253 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fe3bc02fdb0 0x7fe3bc032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:02.140 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.137+0000 7fe3e1b39700 1 -- 172.21.15.139:0/4041042253 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3dc059100 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.141 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.137+0000 7fe3d2ffd700 1 --2- 172.21.15.139:0/4041042253 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fe3bc02fdb0 0x7fe3bc032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe3c40096a0 tx=0x7fe3c4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:02.142 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.141+0000 7fe3d8ff9700 1 -- 172.21.15.139:0/4041042253 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3cc016080 con 0x7fe3dc06fcd0 2021-11-20T05:59:02.279 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.292+0000 7fe3e1b39700 1 -- 172.21.15.139:0/4041042253 --> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3dc1048d0 con 0x7fe3bc02fdb0 2021-11-20T05:59:02.279 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.293+0000 7fe3d8ff9700 1 -- 172.21.15.139:0/4041042253 <== mgr.14114 v2:172.21.15.139:6800/1220659339 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fe3dc1048d0 con 0x7fe3bc02fdb0 2021-11-20T05:59:02.280 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDF3MRPQBm4E6xqSiStjeAOMS5Dh0rqP7cp+aNJcOCJ0Ssvwrw/sDmwga8nKkzL/87vY0tF9SKU1XftEPgEosvXbTKb/9sxFyPyQ5jPjnnI26NO7NR338Dt4vszFmtW2YN0NaqGWgGLU+0xcYIg768vZaEIMOIaWhiAcNPvxCqhjKuaOcNConKsSgDwjYjO/KYdFbN2iGTE90/onuktSOhM/NOSTIzqPyMGXIdiLStjG0rmGBTfJMKdLVY95xx8oa6Is6HDqWF6xxTYAMQ1mMZs2cy2O7UCh2kKiQQSH7yA741CLvUMBJiusnkSXR13r7jUd7Ml28QvhUzBm2VgKZrMWQXtRv9eH81qn+UXgCuBaXCmbZKTDdpnw6ok5cNM55/MA1Kq2Al7zEGA8vfZ1Cx0CWeJibUoG91Ho1nolTH8eSN9gNUeiuRF1Uk/WBSKiA6YHIqqdWTd92RhbwnnSA1m5qDECWq8CZZN5/wK3QarHsAl8swpoiM+r+1HTmvWclU= ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T05:59:02.282 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.294+0000 7fe3e1b39700 1 -- 172.21.15.139:0/4041042253 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fe3bc02fdb0 msgr2=0x7fe3bc032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:02.282 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.294+0000 7fe3e1b39700 1 --2- 172.21.15.139:0/4041042253 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fe3bc02fdb0 0x7fe3bc032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe3c40096a0 tx=0x7fe3c4006b40).stop 2021-11-20T05:59:02.282 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.294+0000 7fe3e1b39700 1 -- 172.21.15.139:0/4041042253 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3dc06fcd0 msgr2=0x7fe3dc10a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:02.282 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.294+0000 7fe3e1b39700 1 --2- 172.21.15.139:0/4041042253 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3dc06fcd0 0x7fe3dc10a9a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fe3cc00e040 tx=0x7fe3cc006ee0).stop 2021-11-20T05:59:02.283 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.294+0000 7fe3e1b39700 1 -- 172.21.15.139:0/4041042253 shutdown_connections 2021-11-20T05:59:02.283 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.294+0000 7fe3e1b39700 1 --2- 172.21.15.139:0/4041042253 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fe3bc02fdb0 0x7fe3bc032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:02.283 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.294+0000 7fe3e1b39700 1 --2- 172.21.15.139:0/4041042253 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3dc06fcd0 0x7fe3dc10a9a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:02.284 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.294+0000 7fe3e1b39700 1 -- 172.21.15.139:0/4041042253 >> 172.21.15.139:0/4041042253 conn(0x7fe3dc0f6280 msgr2=0x7fe3dc0f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:02.284 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.295+0000 7fe3e1b39700 1 -- 172.21.15.139:0/4041042253 shutdown_connections 2021-11-20T05:59:02.284 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:02.295+0000 7fe3e1b39700 1 -- 172.21.15.139:0/4041042253 wait complete. 2021-11-20T05:59:02.639 INFO:teuthology.orchestra.run.smithi139.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2021-11-20T05:59:02.639 INFO:teuthology.orchestra.run.smithi139.stderr:Adding key to root@localhost authorized_keys... 2021-11-20T05:59:02.640 INFO:teuthology.orchestra.run.smithi139.stderr:Adding host smithi139... 2021-11-20T05:59:02.640 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi139 172.21.15.139 2021-11-20T05:59:02.768 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:02 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:01.066705+0000 mgr.smithi139.dplyxn (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2021-11-20T05:59:02.768 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:02 smithi139 conmon[33501]: audit 2021-11-20T05:59:02.769 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:02 smithi139 conmon[33501]: 2021-11-20T05:59:01.412229+0000 mon.smithi139 (mon.0) 51 : audit [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:02.769 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:02 smithi139 conmon[33501]: audit 2021-11-20T05:59:01.417371+0000 mon.smithi139 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:03.117 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.126+0000 7f7bb73e8700 1 Processor -- start 2021-11-20T05:59:03.118 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.126+0000 7f7bb73e8700 1 -- start start 2021-11-20T05:59:03.118 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.126+0000 7f7bb73e8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bb0071cd0 0x7f7bb0070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:03.118 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.126+0000 7f7bb73e8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb00720f0 con 0x7f7bb0071cd0 2021-11-20T05:59:03.118 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.126+0000 7f7bb5184700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bb0071cd0 0x7f7bb0070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:03.119 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.126+0000 7f7bb5184700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bb0071cd0 0x7f7bb0070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56544/0 (socket says 172.21.15.139:56544) 2021-11-20T05:59:03.119 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.126+0000 7f7bb5184700 1 -- 172.21.15.139:0/1110798878 learned_addr learned my addr 172.21.15.139:0/1110798878 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:03.119 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.127+0000 7f7bb5184700 1 -- 172.21.15.139:0/1110798878 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb0070760 con 0x7f7bb0071cd0 2021-11-20T05:59:03.119 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.127+0000 7f7bb5184700 1 --2- 172.21.15.139:0/1110798878 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bb0071cd0 0x7f7bb0070270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f7bac00b5f0 tx=0x7f7bac006ee0).ready entity=mon.0 client_cookie=34d32a7dd3c97be8 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:03.120 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.127+0000 7f7b9ffff700 1 -- 172.21.15.139:0/1110798878 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bac00d600 con 0x7f7bb0071cd0 2021-11-20T05:59:03.120 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.127+0000 7f7b9ffff700 1 -- 172.21.15.139:0/1110798878 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7bac00dbd0 con 0x7f7bb0071cd0 2021-11-20T05:59:03.120 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.127+0000 7f7b9ffff700 1 -- 172.21.15.139:0/1110798878 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bac011c00 con 0x7f7bb0071cd0 2021-11-20T05:59:03.120 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.128+0000 7f7bb73e8700 1 -- 172.21.15.139:0/1110798878 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bb0071cd0 msgr2=0x7f7bb0070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:03.121 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.128+0000 7f7bb73e8700 1 --2- 172.21.15.139:0/1110798878 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bb0071cd0 0x7f7bb0070270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f7bac00b5f0 tx=0x7f7bac006ee0).stop 2021-11-20T05:59:03.121 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.128+0000 7f7bb73e8700 1 -- 172.21.15.139:0/1110798878 shutdown_connections 2021-11-20T05:59:03.123 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.128+0000 7f7bb73e8700 1 --2- 172.21.15.139:0/1110798878 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bb0071cd0 0x7f7bb0070270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:03.123 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.128+0000 7f7bb73e8700 1 -- 172.21.15.139:0/1110798878 >> 172.21.15.139:0/1110798878 conn(0x7f7bb00f73e0 msgr2=0x7f7bb00f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:03.123 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.128+0000 7f7bb73e8700 1 -- 172.21.15.139:0/1110798878 shutdown_connections 2021-11-20T05:59:03.123 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.128+0000 7f7bb73e8700 1 -- 172.21.15.139:0/1110798878 wait complete. 2021-11-20T05:59:03.124 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.128+0000 7f7bb73e8700 1 Processor -- start 2021-11-20T05:59:03.124 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.128+0000 7f7bb73e8700 1 -- start start 2021-11-20T05:59:03.124 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.129+0000 7f7bb73e8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bb0071cd0 0x7f7bb010cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:03.124 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.129+0000 7f7bb73e8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb010d0a0 con 0x7f7bb0071cd0 2021-11-20T05:59:03.125 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.129+0000 7f7bb5184700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bb0071cd0 0x7f7bb010cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:03.125 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.129+0000 7f7bb5184700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bb0071cd0 0x7f7bb010cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56546/0 (socket says 172.21.15.139:56546) 2021-11-20T05:59:03.125 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.129+0000 7f7bb5184700 1 -- 172.21.15.139:0/1130858224 learned_addr learned my addr 172.21.15.139:0/1130858224 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:03.125 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.129+0000 7f7bb5184700 1 -- 172.21.15.139:0/1130858224 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bac00b200 con 0x7f7bb0071cd0 2021-11-20T05:59:03.126 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.129+0000 7f7bb5184700 1 --2- 172.21.15.139:0/1130858224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bb0071cd0 0x7f7bb010cbb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f7bac00e040 tx=0x7f7bac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:03.126 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.129+0000 7f7b9dffb700 1 -- 172.21.15.139:0/1130858224 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bac00d9f0 con 0x7f7bb0071cd0 2021-11-20T05:59:03.126 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.129+0000 7f7b9dffb700 1 -- 172.21.15.139:0/1130858224 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7bac01a9f0 con 0x7f7bb0071cd0 2021-11-20T05:59:03.126 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.129+0000 7f7b9dffb700 1 -- 172.21.15.139:0/1130858224 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bac01b8c0 con 0x7f7bb0071cd0 2021-11-20T05:59:03.127 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.129+0000 7f7bb73e8700 1 -- 172.21.15.139:0/1130858224 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb010d2a0 con 0x7f7bb0071cd0 2021-11-20T05:59:03.127 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.130+0000 7f7bb73e8700 1 -- 172.21.15.139:0/1130858224 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb010d6b0 con 0x7f7bb0071cd0 2021-11-20T05:59:03.127 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.130+0000 7f7b9dffb700 1 -- 172.21.15.139:0/1130858224 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f7bac00f070 con 0x7f7bb0071cd0 2021-11-20T05:59:03.127 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.130+0000 7f7b9dffb700 1 --2- 172.21.15.139:0/1130858224 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f7b9802fdb0 0x7f7b98032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:03.128 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.131+0000 7f7b9dffb700 1 -- 172.21.15.139:0/1130858224 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f7bac0412c0 con 0x7f7bb0071cd0 2021-11-20T05:59:03.128 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.130+0000 7f7bb73e8700 1 -- 172.21.15.139:0/1130858224 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb0045ad0 con 0x7f7bb0071cd0 2021-11-20T05:59:03.128 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.131+0000 7f7bb4983700 1 --2- 172.21.15.139:0/1130858224 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f7b9802fdb0 0x7f7b98032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:03.129 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.131+0000 7f7bb4983700 1 --2- 172.21.15.139:0/1130858224 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f7b9802fdb0 0x7f7b98032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7ba00096a0 tx=0x7f7ba0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:03.129 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.135+0000 7f7b9dffb700 1 -- 172.21.15.139:0/1130858224 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7bac016070 con 0x7f7bb0071cd0 2021-11-20T05:59:03.292 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.305+0000 7f7bb73e8700 1 -- 172.21.15.139:0/1130858224 --> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi139", "addr": "172.21.15.139", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bb0106a90 con 0x7f7b9802fdb0 2021-11-20T05:59:03.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:03 smithi139 conmon[33501]: audit 2021-11-20T05:59:02.293774+0000 mgr.smithi139.dplyxn (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:59:03.981 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: Added host 'smithi139' 2021-11-20T05:59:03.981 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.993+0000 7f7b9dffb700 1 -- 172.21.15.139:0/1130858224 <== mgr.14114 v2:172.21.15.139:6800/1220659339 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f7bb0106a90 con 0x7f7b9802fdb0 2021-11-20T05:59:03.983 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.995+0000 7f7bb73e8700 1 -- 172.21.15.139:0/1130858224 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f7b9802fdb0 msgr2=0x7f7b98032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:03.983 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.995+0000 7f7bb73e8700 1 --2- 172.21.15.139:0/1130858224 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f7b9802fdb0 0x7f7b98032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7ba00096a0 tx=0x7f7ba0006b40).stop 2021-11-20T05:59:03.983 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.995+0000 7f7bb73e8700 1 -- 172.21.15.139:0/1130858224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bb0071cd0 msgr2=0x7f7bb010cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:03.983 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.995+0000 7f7bb73e8700 1 --2- 172.21.15.139:0/1130858224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bb0071cd0 0x7f7bb010cbb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f7bac00e040 tx=0x7f7bac006ee0).stop 2021-11-20T05:59:03.984 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.995+0000 7f7bb73e8700 1 -- 172.21.15.139:0/1130858224 shutdown_connections 2021-11-20T05:59:03.984 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.995+0000 7f7bb73e8700 1 --2- 172.21.15.139:0/1130858224 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f7b9802fdb0 0x7f7b98032270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:03.984 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.995+0000 7f7bb73e8700 1 --2- 172.21.15.139:0/1130858224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bb0071cd0 0x7f7bb010cbb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:03.984 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.995+0000 7f7bb73e8700 1 -- 172.21.15.139:0/1130858224 >> 172.21.15.139:0/1130858224 conn(0x7f7bb00f73e0 msgr2=0x7f7bb00f8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:03.984 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.996+0000 7f7bb73e8700 1 -- 172.21.15.139:0/1130858224 shutdown_connections 2021-11-20T05:59:03.985 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:03.996+0000 7f7bb73e8700 1 -- 172.21.15.139:0/1130858224 wait complete. 2021-11-20T05:59:04.377 INFO:teuthology.orchestra.run.smithi139.stderr:Deploying mon service with default placement... 2021-11-20T05:59:04.377 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2021-11-20T05:59:04.560 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:04 smithi139 conmon[33501]: audit 2021-11-20T05:59:03.307094+0000 2021-11-20T05:59:04.560 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:04 smithi139 conmon[33501]: mgr.smithi139.dplyxn (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi139", "addr": "172.21.15.139", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:59:04.560 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:04 smithi139 conmon[33501]: audit 2021-11-20T05:59:03.993139+0000 mon.smithi139 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:04.561 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:04 smithi139 conmon[33501]: audit 2021-11-20T05:59:03.993703+0000 mon.smithi139 (mon.0) 54 : audit [DBG] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:59:04.853 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.862+0000 7f2245713700 1 Processor -- start 2021-11-20T05:59:04.854 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.862+0000 7f2245713700 1 -- start start 2021-11-20T05:59:04.854 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.862+0000 7f2245713700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f224006fbd0 0x7f2240100af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:04.854 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.862+0000 7f2245713700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22400fc4e0 con 0x7f224006fbd0 2021-11-20T05:59:04.855 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.862+0000 7f223effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f224006fbd0 0x7f2240100af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:04.857 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.862+0000 7f223effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f224006fbd0 0x7f2240100af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56560/0 (socket says 172.21.15.139:56560) 2021-11-20T05:59:04.857 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.862+0000 7f223effd700 1 -- 172.21.15.139:0/1264949257 learned_addr learned my addr 172.21.15.139:0/1264949257 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:04.857 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.862+0000 7f223effd700 1 -- 172.21.15.139:0/1264949257 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2240100fe0 con 0x7f224006fbd0 2021-11-20T05:59:04.857 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.862+0000 7f223effd700 1 --2- 172.21.15.139:0/1264949257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f224006fbd0 0x7f2240100af0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f223000b5f0 tx=0x7f2230006ee0).ready entity=mon.0 client_cookie=c9724d4d389c1e61 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:04.858 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.863+0000 7f223dffb700 1 -- 172.21.15.139:0/1264949257 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f223000d600 con 0x7f224006fbd0 2021-11-20T05:59:04.858 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.863+0000 7f223dffb700 1 -- 172.21.15.139:0/1264949257 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f223000dbd0 con 0x7f224006fbd0 2021-11-20T05:59:04.858 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.863+0000 7f223dffb700 1 -- 172.21.15.139:0/1264949257 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2230011c00 con 0x7f224006fbd0 2021-11-20T05:59:04.858 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.863+0000 7f2245713700 1 -- 172.21.15.139:0/1264949257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f224006fbd0 msgr2=0x7f2240100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:04.859 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.863+0000 7f2245713700 1 --2- 172.21.15.139:0/1264949257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f224006fbd0 0x7f2240100af0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f223000b5f0 tx=0x7f2230006ee0).stop 2021-11-20T05:59:04.859 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.863+0000 7f2245713700 1 -- 172.21.15.139:0/1264949257 shutdown_connections 2021-11-20T05:59:04.859 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.863+0000 7f2245713700 1 --2- 172.21.15.139:0/1264949257 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f224006fbd0 0x7f2240100af0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:04.859 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.863+0000 7f2245713700 1 -- 172.21.15.139:0/1264949257 >> 172.21.15.139:0/1264949257 conn(0x7f22400f6170 msgr2=0x7f22400f8590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:04.859 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.863+0000 7f2245713700 1 -- 172.21.15.139:0/1264949257 shutdown_connections 2021-11-20T05:59:04.860 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.863+0000 7f2245713700 1 -- 172.21.15.139:0/1264949257 wait complete. 2021-11-20T05:59:04.860 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.864+0000 7f2245713700 1 Processor -- start 2021-11-20T05:59:04.860 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.864+0000 7f2245713700 1 -- start start 2021-11-20T05:59:04.860 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.864+0000 7f2245713700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f224010cb40 0x7f224010cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:04.861 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.864+0000 7f2245713700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224010d450 con 0x7f224010cb40 2021-11-20T05:59:04.861 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.864+0000 7f223effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f224010cb40 0x7f224010cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:04.861 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.864+0000 7f223effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f224010cb40 0x7f224010cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56562/0 (socket says 172.21.15.139:56562) 2021-11-20T05:59:04.861 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.864+0000 7f223effd700 1 -- 172.21.15.139:0/2973496594 learned_addr learned my addr 172.21.15.139:0/2973496594 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:04.861 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.865+0000 7f223effd700 1 -- 172.21.15.139:0/2973496594 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223000b200 con 0x7f224010cb40 2021-11-20T05:59:04.862 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.865+0000 7f223effd700 1 --2- 172.21.15.139:0/2973496594 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f224010cb40 0x7f224010cf60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f2230000c00 tx=0x7f2230006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:04.862 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.865+0000 7f22277fe700 1 -- 172.21.15.139:0/2973496594 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2230011a20 con 0x7f224010cb40 2021-11-20T05:59:04.862 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.865+0000 7f2245713700 1 -- 172.21.15.139:0/2973496594 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224010d650 con 0x7f224010cb40 2021-11-20T05:59:04.862 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.865+0000 7f22277fe700 1 -- 172.21.15.139:0/2973496594 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f223001a9f0 con 0x7f224010cb40 2021-11-20T05:59:04.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.865+0000 7f22277fe700 1 -- 172.21.15.139:0/2973496594 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f223001b8c0 con 0x7f224010cb40 2021-11-20T05:59:04.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.865+0000 7f2245713700 1 -- 172.21.15.139:0/2973496594 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224010e120 con 0x7f224010cb40 2021-11-20T05:59:04.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.866+0000 7f22277fe700 1 -- 172.21.15.139:0/2973496594 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f223000f070 con 0x7f224010cb40 2021-11-20T05:59:04.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.866+0000 7f22277fe700 1 --2- 172.21.15.139:0/2973496594 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f222c02fd50 0x7f222c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:04.864 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.866+0000 7f22277fe700 1 -- 172.21.15.139:0/2973496594 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f22300411d0 con 0x7f224010cb40 2021-11-20T05:59:04.864 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.866+0000 7f2245713700 1 -- 172.21.15.139:0/2973496594 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2240059100 con 0x7f224010cb40 2021-11-20T05:59:04.864 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.866+0000 7f223e7fc700 1 --2- 172.21.15.139:0/2973496594 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f222c02fd50 0x7f222c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:04.864 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.867+0000 7f223e7fc700 1 --2- 172.21.15.139:0/2973496594 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f222c02fd50 0x7f222c032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f22340096a0 tx=0x7f2234006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:04.865 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:04.870+0000 7f22277fe700 1 -- 172.21.15.139:0/2973496594 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2230025460 con 0x7f224010cb40 2021-11-20T05:59:05.037 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.049+0000 7f2245713700 1 -- 172.21.15.139:0/2973496594 --> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f22401064b0 con 0x7f222c02fd50 2021-11-20T05:59:05.041 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.053+0000 7f22277fe700 1 -- 172.21.15.139:0/2973496594 <== mgr.14114 v2:172.21.15.139:6800/1220659339 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f22401064b0 con 0x7f222c02fd50 2021-11-20T05:59:05.041 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: Scheduled mon update... 2021-11-20T05:59:05.043 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.055+0000 7f2245713700 1 -- 172.21.15.139:0/2973496594 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f222c02fd50 msgr2=0x7f222c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:05.043 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.055+0000 7f2245713700 1 --2- 172.21.15.139:0/2973496594 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f222c02fd50 0x7f222c032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f22340096a0 tx=0x7f2234006b40).stop 2021-11-20T05:59:05.044 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.055+0000 7f2245713700 1 -- 172.21.15.139:0/2973496594 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f224010cb40 msgr2=0x7f224010cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:05.044 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.055+0000 7f2245713700 1 --2- 172.21.15.139:0/2973496594 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f224010cb40 0x7f224010cf60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f2230000c00 tx=0x7f2230006ee0).stop 2021-11-20T05:59:05.044 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.055+0000 7f2245713700 1 -- 172.21.15.139:0/2973496594 shutdown_connections 2021-11-20T05:59:05.044 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.055+0000 7f2245713700 1 --2- 172.21.15.139:0/2973496594 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f222c02fd50 0x7f222c032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:05.045 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.055+0000 7f2245713700 1 --2- 172.21.15.139:0/2973496594 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f224010cb40 0x7f224010cf60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:05.045 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.055+0000 7f2245713700 1 -- 172.21.15.139:0/2973496594 >> 172.21.15.139:0/2973496594 conn(0x7f22400f6170 msgr2=0x7f22400f7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:05.045 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.055+0000 7f2245713700 1 -- 172.21.15.139:0/2973496594 shutdown_connections 2021-11-20T05:59:05.045 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.055+0000 7f2245713700 1 -- 172.21.15.139:0/2973496594 wait complete. 2021-11-20T05:59:05.445 INFO:teuthology.orchestra.run.smithi139.stderr:Deploying mgr service with default placement... 2021-11-20T05:59:05.445 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2021-11-20T05:59:05.447 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:05 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:03.993448+0000 mgr.smithi139.dplyxn (mgr.14114) 9 : cephadm [INF] Added host smithi139 2021-11-20T05:59:05.447 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:05 smithi139 conmon[33501]: audit 2021-11-20T05:59:05.054877+0000 mon.smithi139 (mon.0) 55 : audit [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:05.945 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.952+0000 7fb842195700 1 Processor -- start 2021-11-20T05:59:05.945 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.952+0000 7fb842195700 1 -- start start 2021-11-20T05:59:05.945 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.952+0000 7fb842195700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb83c0faae0 0x7fb83c0faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:05.945 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.952+0000 7fb842195700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c0fb480 con 0x7fb83c0faae0 2021-11-20T05:59:05.946 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.953+0000 7fb83b7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb83c0faae0 0x7fb83c0faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:05.946 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.953+0000 7fb83b7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb83c0faae0 0x7fb83c0faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56566/0 (socket says 172.21.15.139:56566) 2021-11-20T05:59:05.946 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.953+0000 7fb83b7fe700 1 -- 172.21.15.139:0/2380914170 learned_addr learned my addr 172.21.15.139:0/2380914170 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:05.946 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.953+0000 7fb83b7fe700 1 -- 172.21.15.139:0/2380914170 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c0fb5c0 con 0x7fb83c0faae0 2021-11-20T05:59:05.947 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.953+0000 7fb83b7fe700 1 --2- 172.21.15.139:0/2380914170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb83c0faae0 0x7fb83c0faf00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fb82c00b5f0 tx=0x7fb82c006ee0).ready entity=mon.0 client_cookie=d1515c7da3dc5bf9 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:05.954 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.953+0000 7fb83affd700 1 -- 172.21.15.139:0/2380914170 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb82c00d600 con 0x7fb83c0faae0 2021-11-20T05:59:05.954 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.953+0000 7fb83affd700 1 -- 172.21.15.139:0/2380914170 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb82c00dbd0 con 0x7fb83c0faae0 2021-11-20T05:59:05.955 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.953+0000 7fb83affd700 1 -- 172.21.15.139:0/2380914170 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb82c011c00 con 0x7fb83c0faae0 2021-11-20T05:59:05.955 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.954+0000 7fb842195700 1 -- 172.21.15.139:0/2380914170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb83c0faae0 msgr2=0x7fb83c0faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:05.956 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.954+0000 7fb842195700 1 --2- 172.21.15.139:0/2380914170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb83c0faae0 0x7fb83c0faf00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fb82c00b5f0 tx=0x7fb82c006ee0).stop 2021-11-20T05:59:05.956 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.954+0000 7fb842195700 1 -- 172.21.15.139:0/2380914170 shutdown_connections 2021-11-20T05:59:05.956 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.954+0000 7fb842195700 1 --2- 172.21.15.139:0/2380914170 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb83c0faae0 0x7fb83c0faf00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:05.956 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.954+0000 7fb842195700 1 -- 172.21.15.139:0/2380914170 >> 172.21.15.139:0/2380914170 conn(0x7fb83c0f6280 msgr2=0x7fb83c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:05.957 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.954+0000 7fb842195700 1 -- 172.21.15.139:0/2380914170 shutdown_connections 2021-11-20T05:59:05.957 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.954+0000 7fb842195700 1 -- 172.21.15.139:0/2380914170 wait complete. 2021-11-20T05:59:05.957 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.955+0000 7fb842195700 1 Processor -- start 2021-11-20T05:59:05.957 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.955+0000 7fb842195700 1 -- start start 2021-11-20T05:59:05.958 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.955+0000 7fb842195700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb83c0faae0 0x7fb83c109930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:05.958 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.955+0000 7fb842195700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c109e20 con 0x7fb83c0faae0 2021-11-20T05:59:05.958 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.955+0000 7fb83b7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb83c0faae0 0x7fb83c109930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:05.958 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.955+0000 7fb83b7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb83c0faae0 0x7fb83c109930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56568/0 (socket says 172.21.15.139:56568) 2021-11-20T05:59:05.958 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.955+0000 7fb83b7fe700 1 -- 172.21.15.139:0/17047115 learned_addr learned my addr 172.21.15.139:0/17047115 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:05.959 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.956+0000 7fb83b7fe700 1 -- 172.21.15.139:0/17047115 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c00b200 con 0x7fb83c0faae0 2021-11-20T05:59:05.959 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.956+0000 7fb83b7fe700 1 --2- 172.21.15.139:0/17047115 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb83c0faae0 0x7fb83c109930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fb82c002f40 tx=0x7fb82c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:05.959 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.956+0000 7fb838ff9700 1 -- 172.21.15.139:0/17047115 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb82c011690 con 0x7fb83c0faae0 2021-11-20T05:59:05.959 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.956+0000 7fb838ff9700 1 -- 172.21.15.139:0/17047115 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb82c01b880 con 0x7fb83c0faae0 2021-11-20T05:59:05.960 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.956+0000 7fb842195700 1 -- 172.21.15.139:0/17047115 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83c10a020 con 0x7fb83c0faae0 2021-11-20T05:59:05.960 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.956+0000 7fb838ff9700 1 -- 172.21.15.139:0/17047115 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb82c01a6b0 con 0x7fb83c0faae0 2021-11-20T05:59:05.960 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.956+0000 7fb842195700 1 -- 172.21.15.139:0/17047115 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83c10a430 con 0x7fb83c0faae0 2021-11-20T05:59:05.962 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.956+0000 7fb838ff9700 1 -- 172.21.15.139:0/17047115 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb82c00f070 con 0x7fb83c0faae0 2021-11-20T05:59:05.962 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.957+0000 7fb838ff9700 1 --2- 172.21.15.139:0/17047115 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fb81c031f60 0x7fb81c034420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:05.962 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.957+0000 7fb838ff9700 1 -- 172.21.15.139:0/17047115 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb82c042030 con 0x7fb83c0faae0 2021-11-20T05:59:05.962 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.957+0000 7fb832ffd700 1 --2- 172.21.15.139:0/17047115 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fb81c031f60 0x7fb81c034420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:05.965 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.957+0000 7fb832ffd700 1 --2- 172.21.15.139:0/17047115 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fb81c031f60 0x7fb81c034420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fb8240096a0 tx=0x7fb824006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:05.965 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.957+0000 7fb842195700 1 -- 172.21.15.139:0/17047115 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb820004fa0 con 0x7fb83c0faae0 2021-11-20T05:59:05.965 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:05.962+0000 7fb838ff9700 1 -- 172.21.15.139:0/17047115 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb82c016070 con 0x7fb83c0faae0 2021-11-20T05:59:06.138 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.150+0000 7fb842195700 1 -- 172.21.15.139:0/17047115 --> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fb820002a50 con 0x7fb81c031f60 2021-11-20T05:59:06.142 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.154+0000 7fb838ff9700 1 -- 172.21.15.139:0/17047115 <== mgr.14114 v2:172.21.15.139:6800/1220659339 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb820002a50 con 0x7fb81c031f60 2021-11-20T05:59:06.142 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: Scheduled mgr update... 2021-11-20T05:59:06.144 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.156+0000 7fb842195700 1 -- 172.21.15.139:0/17047115 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fb81c031f60 msgr2=0x7fb81c034420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:06.145 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.156+0000 7fb842195700 1 --2- 172.21.15.139:0/17047115 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fb81c031f60 0x7fb81c034420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fb8240096a0 tx=0x7fb824006b40).stop 2021-11-20T05:59:06.145 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.156+0000 7fb842195700 1 -- 172.21.15.139:0/17047115 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb83c0faae0 msgr2=0x7fb83c109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:06.145 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.156+0000 7fb842195700 1 --2- 172.21.15.139:0/17047115 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb83c0faae0 0x7fb83c109930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fb82c002f40 tx=0x7fb82c006ee0).stop 2021-11-20T05:59:06.145 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.156+0000 7fb842195700 1 -- 172.21.15.139:0/17047115 shutdown_connections 2021-11-20T05:59:06.146 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.156+0000 7fb842195700 1 --2- 172.21.15.139:0/17047115 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fb81c031f60 0x7fb81c034420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:06.146 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.156+0000 7fb842195700 1 --2- 172.21.15.139:0/17047115 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb83c0faae0 0x7fb83c109930 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:06.146 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.156+0000 7fb842195700 1 -- 172.21.15.139:0/17047115 >> 172.21.15.139:0/17047115 conn(0x7fb83c0f6280 msgr2=0x7fb83c0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:06.146 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.157+0000 7fb842195700 1 -- 172.21.15.139:0/17047115 shutdown_connections 2021-11-20T05:59:06.147 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.157+0000 7fb842195700 1 -- 172.21.15.139:0/17047115 wait complete. 2021-11-20T05:59:06.430 INFO:teuthology.orchestra.run.smithi139.stderr:Deploying crash service with default placement... 2021-11-20T05:59:06.430 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2021-11-20T05:59:06.670 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:06 smithi139 conmon[33501]: audit 2021-11-20T05:59:05.051847+0000 mgr.smithi139.dplyxn 2021-11-20T05:59:06.671 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:06 smithi139 conmon[33501]: (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:59:06.671 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:06 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:05.052533+0000 mgr.smithi139.dplyxn (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2021-11-20T05:59:06.672 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:06 smithi139 conmon[33501]: audit 2021-11-20T05:59:06.156031+0000 mon.smithi139 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:06.672 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:06 smithi139 conmon[33501]: audit 2021-11-20T05:59:06.220011+0000 mon.smithi139 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:06.894 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.902+0000 7f0bb218a700 1 Processor -- start 2021-11-20T05:59:06.895 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.902+0000 7f0bb218a700 1 -- start start 2021-11-20T05:59:06.895 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.902+0000 7f0bb218a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0bac0fc0c0 0x7f0bac0fc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:06.895 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.902+0000 7f0bb218a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac0fc9d0 con 0x7f0bac0fc0c0 2021-11-20T05:59:06.895 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.902+0000 7f0bab7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0bac0fc0c0 0x7f0bac0fc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:06.896 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.902+0000 7f0bab7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0bac0fc0c0 0x7f0bac0fc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56572/0 (socket says 172.21.15.139:56572) 2021-11-20T05:59:06.896 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.902+0000 7f0bab7fe700 1 -- 172.21.15.139:0/140725961 learned_addr learned my addr 172.21.15.139:0/140725961 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:06.896 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.903+0000 7f0bab7fe700 1 -- 172.21.15.139:0/140725961 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac0fcb10 con 0x7f0bac0fc0c0 2021-11-20T05:59:06.896 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.903+0000 7f0bab7fe700 1 --2- 172.21.15.139:0/140725961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0bac0fc0c0 0x7f0bac0fc4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f0b9c00b870 tx=0x7f0b9c006f70).ready entity=mon.0 client_cookie=30eb0838e92640f4 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:06.897 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.903+0000 7f0baa7fc700 1 -- 172.21.15.139:0/140725961 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9c00da20 con 0x7f0bac0fc0c0 2021-11-20T05:59:06.897 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.903+0000 7f0baa7fc700 1 -- 172.21.15.139:0/140725961 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b9c010470 con 0x7f0bac0fc0c0 2021-11-20T05:59:06.897 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.903+0000 7f0baa7fc700 1 -- 172.21.15.139:0/140725961 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9c00f5b0 con 0x7f0bac0fc0c0 2021-11-20T05:59:06.897 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.904+0000 7f0bb218a700 1 -- 172.21.15.139:0/140725961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0bac0fc0c0 msgr2=0x7f0bac0fc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:06.898 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.904+0000 7f0bb218a700 1 --2- 172.21.15.139:0/140725961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0bac0fc0c0 0x7f0bac0fc4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f0b9c00b870 tx=0x7f0b9c006f70).stop 2021-11-20T05:59:06.898 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.904+0000 7f0bb218a700 1 -- 172.21.15.139:0/140725961 shutdown_connections 2021-11-20T05:59:06.898 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.904+0000 7f0bb218a700 1 --2- 172.21.15.139:0/140725961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0bac0fc0c0 0x7f0bac0fc4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:06.898 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.904+0000 7f0bb218a700 1 -- 172.21.15.139:0/140725961 >> 172.21.15.139:0/140725961 conn(0x7f0bac0f5a80 msgr2=0x7f0bac0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:06.898 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.904+0000 7f0bb218a700 1 -- 172.21.15.139:0/140725961 shutdown_connections 2021-11-20T05:59:06.899 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.904+0000 7f0bb218a700 1 -- 172.21.15.139:0/140725961 wait complete. 2021-11-20T05:59:06.899 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.904+0000 7f0bb218a700 1 Processor -- start 2021-11-20T05:59:06.899 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.904+0000 7f0bb218a700 1 -- start start 2021-11-20T05:59:06.900 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.905+0000 7f0bb218a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0bac0fc0c0 0x7f0bac10cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:06.900 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.905+0000 7f0bb218a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac10d0e0 con 0x7f0bac0fc0c0 2021-11-20T05:59:06.900 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.905+0000 7f0bab7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0bac0fc0c0 0x7f0bac10cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:06.901 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.905+0000 7f0bab7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0bac0fc0c0 0x7f0bac10cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56574/0 (socket says 172.21.15.139:56574) 2021-11-20T05:59:06.902 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.905+0000 7f0bab7fe700 1 -- 172.21.15.139:0/2844522638 learned_addr learned my addr 172.21.15.139:0/2844522638 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:06.902 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.905+0000 7f0bab7fe700 1 -- 172.21.15.139:0/2844522638 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c00b480 con 0x7f0bac0fc0c0 2021-11-20T05:59:06.902 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.905+0000 7f0bab7fe700 1 --2- 172.21.15.139:0/2844522638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0bac0fc0c0 0x7f0bac10cbf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f0b9c006800 tx=0x7f0b9c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:06.902 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.905+0000 7f0b93fff700 1 -- 172.21.15.139:0/2844522638 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9c00f9f0 con 0x7f0bac0fc0c0 2021-11-20T05:59:06.903 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.905+0000 7f0b93fff700 1 -- 172.21.15.139:0/2844522638 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b9c019450 con 0x7f0bac0fc0c0 2021-11-20T05:59:06.903 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.905+0000 7f0bb218a700 1 -- 172.21.15.139:0/2844522638 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bac10d2e0 con 0x7f0bac0fc0c0 2021-11-20T05:59:06.903 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.905+0000 7f0b93fff700 1 -- 172.21.15.139:0/2844522638 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9c023620 con 0x7f0bac0fc0c0 2021-11-20T05:59:06.903 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.905+0000 7f0bb218a700 1 -- 172.21.15.139:0/2844522638 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bac10d6f0 con 0x7f0bac0fc0c0 2021-11-20T05:59:06.904 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.906+0000 7f0b93fff700 1 -- 172.21.15.139:0/2844522638 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0b9c016030 con 0x7f0bac0fc0c0 2021-11-20T05:59:06.904 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.906+0000 7f0b93fff700 1 --2- 172.21.15.139:0/2844522638 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f0b9802fd50 0x7f0b98032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:06.904 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.906+0000 7f0b93fff700 1 -- 172.21.15.139:0/2844522638 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0b9c041d70 con 0x7f0bac0fc0c0 2021-11-20T05:59:06.904 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.906+0000 7f0bb218a700 1 -- 172.21.15.139:0/2844522638 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bac045ad0 con 0x7f0bac0fc0c0 2021-11-20T05:59:06.905 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.906+0000 7f0baaffd700 1 --2- 172.21.15.139:0/2844522638 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f0b9802fd50 0x7f0b98032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:06.905 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.907+0000 7f0baaffd700 1 --2- 172.21.15.139:0/2844522638 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f0b9802fd50 0x7f0b98032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0ba00096a0 tx=0x7f0ba0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:06.908 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:06.911+0000 7f0b93fff700 1 -- 172.21.15.139:0/2844522638 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b9c014020 con 0x7f0bac0fc0c0 2021-11-20T05:59:07.081 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.094+0000 7f0bb218a700 1 -- 172.21.15.139:0/2844522638 --> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bac1026e0 con 0x7f0b9802fd50 2021-11-20T05:59:07.086 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.099+0000 7f0b93fff700 1 -- 172.21.15.139:0/2844522638 <== mgr.14114 v2:172.21.15.139:6800/1220659339 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f0bac1026e0 con 0x7f0b9802fd50 2021-11-20T05:59:07.086 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: Scheduled crash update... 2021-11-20T05:59:07.089 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.101+0000 7f0bb218a700 1 -- 172.21.15.139:0/2844522638 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f0b9802fd50 msgr2=0x7f0b98032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:07.089 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.101+0000 7f0bb218a700 1 --2- 172.21.15.139:0/2844522638 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f0b9802fd50 0x7f0b98032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0ba00096a0 tx=0x7f0ba0006b40).stop 2021-11-20T05:59:07.089 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.101+0000 7f0bb218a700 1 -- 172.21.15.139:0/2844522638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0bac0fc0c0 msgr2=0x7f0bac10cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:07.090 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.101+0000 7f0bb218a700 1 --2- 172.21.15.139:0/2844522638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0bac0fc0c0 0x7f0bac10cbf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f0b9c006800 tx=0x7f0b9c006f70).stop 2021-11-20T05:59:07.090 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.101+0000 7f0bb218a700 1 -- 172.21.15.139:0/2844522638 shutdown_connections 2021-11-20T05:59:07.090 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.101+0000 7f0bb218a700 1 --2- 172.21.15.139:0/2844522638 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f0b9802fd50 0x7f0b98032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:07.090 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.101+0000 7f0bb218a700 1 --2- 172.21.15.139:0/2844522638 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0bac0fc0c0 0x7f0bac10cbf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:07.091 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.101+0000 7f0bb218a700 1 -- 172.21.15.139:0/2844522638 >> 172.21.15.139:0/2844522638 conn(0x7f0bac0f5a80 msgr2=0x7f0bac0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:07.091 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.101+0000 7f0bb218a700 1 -- 172.21.15.139:0/2844522638 shutdown_connections 2021-11-20T05:59:07.091 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.101+0000 7f0bb218a700 1 -- 172.21.15.139:0/2844522638 wait complete. 2021-11-20T05:59:07.364 INFO:teuthology.orchestra.run.smithi139.stderr:Enabling mgr prometheus module... 2021-11-20T05:59:07.364 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2021-11-20T05:59:07.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:07 smithi139 conmon[33501]: audit 2021-11-20T05:59:06.153206+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T05:59:07.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:07 smithi139 conmon[33501]: 14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:59:07.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:07 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:06.153729+0000 mgr.smithi139.dplyxn (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2021-11-20T05:59:07.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:07 smithi139 conmon[33501]: audit 2021-11-20T05:59:06.594956+0000 mon.smithi139 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:07.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:07 smithi139 conmon[33501]: audit 2021-11-20T05:59:07.100454+0000 mon.smithi139 (mon.0) 59 : audit [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:07.940 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.870+0000 7f32760da700 1 Processor -- start 2021-11-20T05:59:07.940 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.870+0000 7f32760da700 1 -- start start 2021-11-20T05:59:07.940 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.870+0000 7f32760da700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f327006fc90 0x7f32700fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:07.941 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.871+0000 7f32760da700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32700fed60 con 0x7f327006fc90 2021-11-20T05:59:07.941 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.871+0000 7f326f7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f327006fc90 0x7f32700fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:07.941 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.871+0000 7f326f7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f327006fc90 0x7f32700fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56578/0 (socket says 172.21.15.139:56578) 2021-11-20T05:59:07.941 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.871+0000 7f326f7fe700 1 -- 172.21.15.139:0/2263735830 learned_addr learned my addr 172.21.15.139:0/2263735830 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:07.942 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.871+0000 7f326f7fe700 1 -- 172.21.15.139:0/2263735830 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32700feea0 con 0x7f327006fc90 2021-11-20T05:59:07.942 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.871+0000 7f326f7fe700 1 --2- 172.21.15.139:0/2263735830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f327006fc90 0x7f32700fe7e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f326000b5f0 tx=0x7f3260006ee0).ready entity=mon.0 client_cookie=329596222b118303 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:07.942 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.871+0000 7f326e7fc700 1 -- 172.21.15.139:0/2263735830 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f326000d600 con 0x7f327006fc90 2021-11-20T05:59:07.942 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.871+0000 7f326e7fc700 1 -- 172.21.15.139:0/2263735830 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f326000dc10 con 0x7f327006fc90 2021-11-20T05:59:07.943 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.871+0000 7f326e7fc700 1 -- 172.21.15.139:0/2263735830 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3260011cd0 con 0x7f327006fc90 2021-11-20T05:59:07.943 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.872+0000 7f32760da700 1 -- 172.21.15.139:0/2263735830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f327006fc90 msgr2=0x7f32700fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:07.943 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.872+0000 7f32760da700 1 --2- 172.21.15.139:0/2263735830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f327006fc90 0x7f32700fe7e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f326000b5f0 tx=0x7f3260006ee0).stop 2021-11-20T05:59:07.943 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.872+0000 7f32760da700 1 -- 172.21.15.139:0/2263735830 shutdown_connections 2021-11-20T05:59:07.944 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.872+0000 7f32760da700 1 --2- 172.21.15.139:0/2263735830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f327006fc90 0x7f32700fe7e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:07.944 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.872+0000 7f32760da700 1 -- 172.21.15.139:0/2263735830 >> 172.21.15.139:0/2263735830 conn(0x7f32700f6280 msgr2=0x7f32700f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:07.944 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.872+0000 7f32760da700 1 -- 172.21.15.139:0/2263735830 shutdown_connections 2021-11-20T05:59:07.944 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.872+0000 7f32760da700 1 -- 172.21.15.139:0/2263735830 wait complete. 2021-11-20T05:59:07.944 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.873+0000 7f32760da700 1 Processor -- start 2021-11-20T05:59:07.945 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.873+0000 7f32760da700 1 -- start start 2021-11-20T05:59:07.945 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.873+0000 7f32760da700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f327006fc90 0x7f327010aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:07.945 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.873+0000 7f32760da700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327010af20 con 0x7f327006fc90 2021-11-20T05:59:07.945 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.873+0000 7f326f7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f327006fc90 0x7f327010aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:07.946 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.873+0000 7f326f7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f327006fc90 0x7f327010aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56580/0 (socket says 172.21.15.139:56580) 2021-11-20T05:59:07.946 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.873+0000 7f326f7fe700 1 -- 172.21.15.139:0/1937789836 learned_addr learned my addr 172.21.15.139:0/1937789836 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:07.946 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.873+0000 7f326f7fe700 1 -- 172.21.15.139:0/1937789836 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326000b200 con 0x7f327006fc90 2021-11-20T05:59:07.946 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.873+0000 7f326f7fe700 1 --2- 172.21.15.139:0/1937789836 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f327006fc90 0x7f327010aa30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f326000dfb0 tx=0x7f3260006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:07.948 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.873+0000 7f3257fff700 1 -- 172.21.15.139:0/1937789836 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3260011df0 con 0x7f327006fc90 2021-11-20T05:59:07.949 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.874+0000 7f3257fff700 1 -- 172.21.15.139:0/1937789836 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f326001b8c0 con 0x7f327006fc90 2021-11-20T05:59:07.949 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.874+0000 7f32760da700 1 -- 172.21.15.139:0/1937789836 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327010b120 con 0x7f327006fc90 2021-11-20T05:59:07.949 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.874+0000 7f3257fff700 1 -- 172.21.15.139:0/1937789836 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f326001a7c0 con 0x7f327006fc90 2021-11-20T05:59:07.949 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.874+0000 7f32760da700 1 -- 172.21.15.139:0/1937789836 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f327010b530 con 0x7f327006fc90 2021-11-20T05:59:07.950 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.874+0000 7f3257fff700 1 -- 172.21.15.139:0/1937789836 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f326000f070 con 0x7f327006fc90 2021-11-20T05:59:07.950 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.874+0000 7f3257fff700 1 --2- 172.21.15.139:0/1937789836 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f325c02fd50 0x7f325c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:07.950 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.874+0000 7f3257fff700 1 -- 172.21.15.139:0/1937789836 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f3260041d70 con 0x7f327006fc90 2021-11-20T05:59:07.950 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.874+0000 7f326effd700 1 --2- 172.21.15.139:0/1937789836 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f325c02fd50 0x7f325c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:07.951 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.875+0000 7f32760da700 1 -- 172.21.15.139:0/1937789836 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3270059100 con 0x7f327006fc90 2021-11-20T05:59:07.951 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.875+0000 7f326effd700 1 --2- 172.21.15.139:0/1937789836 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f325c02fd50 0x7f325c032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f32640096a0 tx=0x7f3264006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:07.951 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:07.879+0000 7f3257fff700 1 -- 172.21.15.139:0/1937789836 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3260016070 con 0x7f327006fc90 2021-11-20T05:59:08.066 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:08.079+0000 7f32760da700 1 -- 172.21.15.139:0/1937789836 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f32700ffc40 con 0x7f327006fc90 2021-11-20T05:59:08.583 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:08.596+0000 7f3257fff700 1 -- 172.21.15.139:0/1937789836 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f3260024790 con 0x7f327006fc90 2021-11-20T05:59:08.585 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:08.597+0000 7f3257fff700 1 -- 172.21.15.139:0/1937789836 <== mon.0 v2:172.21.15.139:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v8) v1 ==== 89+0+0 (secure 0 0 0) 0x7f326002ebe0 con 0x7f327006fc90 2021-11-20T05:59:08.587 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:08.599+0000 7f32760da700 1 -- 172.21.15.139:0/1937789836 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f325c02fd50 msgr2=0x7f325c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:08.588 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:08.599+0000 7f32760da700 1 --2- 172.21.15.139:0/1937789836 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f325c02fd50 0x7f325c032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f32640096a0 tx=0x7f3264006b40).stop 2021-11-20T05:59:08.588 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:08.599+0000 7f32760da700 1 -- 172.21.15.139:0/1937789836 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f327006fc90 msgr2=0x7f327010aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:08.588 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:08.599+0000 7f32760da700 1 --2- 172.21.15.139:0/1937789836 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f327006fc90 0x7f327010aa30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f326000dfb0 tx=0x7f3260006ee0).stop 2021-11-20T05:59:08.588 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:08.599+0000 7f32760da700 1 -- 172.21.15.139:0/1937789836 shutdown_connections 2021-11-20T05:59:08.589 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:08.599+0000 7f32760da700 1 --2- 172.21.15.139:0/1937789836 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7f325c02fd50 0x7f325c032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:08.589 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:08.599+0000 7f32760da700 1 --2- 172.21.15.139:0/1937789836 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f327006fc90 0x7f327010aa30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:08.589 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:08.599+0000 7f32760da700 1 -- 172.21.15.139:0/1937789836 >> 172.21.15.139:0/1937789836 conn(0x7f32700f6280 msgr2=0x7f32700f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:08.590 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:08.600+0000 7f32760da700 1 -- 172.21.15.139:0/1937789836 shutdown_connections 2021-11-20T05:59:08.590 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:08.600+0000 7f32760da700 1 -- 172.21.15.139:0/1937789836 wait complete. 2021-11-20T05:59:08.746 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:08 smithi139 conmon[33501]: audit 2021-11-20T05:59:07.096663+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T05:59:08.747 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:08 smithi139 conmon[33501]: 14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:59:08.747 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:08 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:07.097329+0000 mgr.smithi139.dplyxn (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2021-11-20T05:59:08.747 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:08 smithi139 conmon[33501]: audit 2021-11-20T05:59:08.081290+0000 mon.smithi139 (mon.0) 60 : audit [INF] from='client.? 172.21.15.139:0/1937789836' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2021-11-20T05:59:08.748 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:08 smithi139 conmon[33501]: audit 2021-11-20T05:59:08.088339+0000 mon.smithi139 (mon.0) 61 : audit [INF] from='mgr.14114 172.21.15.139:0/2515273483' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:09.234 INFO:teuthology.orchestra.run.smithi139.stderr:Deploying prometheus service with default placement... 2021-11-20T05:59:09.234 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2021-11-20T05:59:09.705 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.714+0000 7fd99b59e700 1 Processor -- start 2021-11-20T05:59:09.705 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.714+0000 7fd99b59e700 1 -- start start 2021-11-20T05:59:09.705 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.714+0000 7fd99b59e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd99c071cd0 0x7fd99c070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:09.706 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.714+0000 7fd99b59e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c0720f0 con 0x7fd99c071cd0 2021-11-20T05:59:09.706 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.714+0000 7fd99a59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd99c071cd0 0x7fd99c070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:09.706 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.714+0000 7fd99a59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd99c071cd0 0x7fd99c070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56594/0 (socket says 172.21.15.139:56594) 2021-11-20T05:59:09.706 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.714+0000 7fd99a59c700 1 -- 172.21.15.139:0/3397187630 learned_addr learned my addr 172.21.15.139:0/3397187630 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:09.707 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.714+0000 7fd99a59c700 1 -- 172.21.15.139:0/3397187630 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99c070760 con 0x7fd99c071cd0 2021-11-20T05:59:09.707 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.715+0000 7fd99a59c700 1 --2- 172.21.15.139:0/3397187630 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd99c071cd0 0x7fd99c070270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd98c00b5f0 tx=0x7fd98c006ee0).ready entity=mon.0 client_cookie=61a4e6c51bc33059 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:09.707 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.715+0000 7fd99959a700 1 -- 172.21.15.139:0/3397187630 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd98c00d600 con 0x7fd99c071cd0 2021-11-20T05:59:09.707 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.715+0000 7fd99959a700 1 -- 172.21.15.139:0/3397187630 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd98c00dc10 con 0x7fd99c071cd0 2021-11-20T05:59:09.708 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.715+0000 7fd99959a700 1 -- 172.21.15.139:0/3397187630 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd98c011cd0 con 0x7fd99c071cd0 2021-11-20T05:59:09.708 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.715+0000 7fd99b59e700 1 -- 172.21.15.139:0/3397187630 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd99c071cd0 msgr2=0x7fd99c070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:09.708 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.715+0000 7fd99b59e700 1 --2- 172.21.15.139:0/3397187630 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd99c071cd0 0x7fd99c070270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd98c00b5f0 tx=0x7fd98c006ee0).stop 2021-11-20T05:59:09.709 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.716+0000 7fd99b59e700 1 -- 172.21.15.139:0/3397187630 shutdown_connections 2021-11-20T05:59:09.709 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.716+0000 7fd99b59e700 1 --2- 172.21.15.139:0/3397187630 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd99c071cd0 0x7fd99c070270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:09.710 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.716+0000 7fd99b59e700 1 -- 172.21.15.139:0/3397187630 >> 172.21.15.139:0/3397187630 conn(0x7fd99c0f71e0 msgr2=0x7fd99c0f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:09.710 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.716+0000 7fd99b59e700 1 -- 172.21.15.139:0/3397187630 shutdown_connections 2021-11-20T05:59:09.710 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.716+0000 7fd99b59e700 1 -- 172.21.15.139:0/3397187630 wait complete. 2021-11-20T05:59:09.711 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.716+0000 7fd99b59e700 1 Processor -- start 2021-11-20T05:59:09.711 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.716+0000 7fd99b59e700 1 -- start start 2021-11-20T05:59:09.712 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.716+0000 7fd99b59e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd99c071cd0 0x7fd99c10c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:09.713 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.716+0000 7fd99b59e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c10cec0 con 0x7fd99c071cd0 2021-11-20T05:59:09.713 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.717+0000 7fd99a59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd99c071cd0 0x7fd99c10c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:09.713 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.717+0000 7fd99a59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd99c071cd0 0x7fd99c10c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56596/0 (socket says 172.21.15.139:56596) 2021-11-20T05:59:09.713 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.717+0000 7fd99a59c700 1 -- 172.21.15.139:0/3555948303 learned_addr learned my addr 172.21.15.139:0/3555948303 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:09.714 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.717+0000 7fd99a59c700 1 -- 172.21.15.139:0/3555948303 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd98c00b200 con 0x7fd99c071cd0 2021-11-20T05:59:09.714 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.717+0000 7fd99a59c700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd99c071cd0 0x7fd99c10c9d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd98c00b1d0 tx=0x7fd98c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:09.714 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.717+0000 7fd992ffd700 1 -- 172.21.15.139:0/3555948303 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd98c011df0 con 0x7fd99c071cd0 2021-11-20T05:59:09.714 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.717+0000 7fd99b59e700 1 -- 172.21.15.139:0/3555948303 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd99c10d0c0 con 0x7fd99c071cd0 2021-11-20T05:59:09.715 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.717+0000 7fd992ffd700 1 -- 172.21.15.139:0/3555948303 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd98c01b8c0 con 0x7fd99c071cd0 2021-11-20T05:59:09.715 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.717+0000 7fd992ffd700 1 -- 172.21.15.139:0/3555948303 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd98c01a830 con 0x7fd99c071cd0 2021-11-20T05:59:09.715 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.717+0000 7fd99b59e700 1 -- 172.21.15.139:0/3555948303 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd99c10d4d0 con 0x7fd99c071cd0 2021-11-20T05:59:09.716 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.718+0000 7fd992ffd700 1 -- 172.21.15.139:0/3555948303 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fd98c00f070 con 0x7fd99c071cd0 2021-11-20T05:59:09.717 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.718+0000 7fd992ffd700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fd97c02fdb0 0x7fd97c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:09.717 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.718+0000 7fd992ffd700 1 -- 172.21.15.139:0/3555948303 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd98c040ef0 con 0x7fd99c071cd0 2021-11-20T05:59:09.718 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.718+0000 7fd999d9b700 1 -- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fd97c02fdb0 msgr2=0x7fd97c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/1220659339 2021-11-20T05:59:09.718 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.718+0000 7fd999d9b700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fd97c02fdb0 0x7fd97c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T05:59:09.718 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.719+0000 7fd99b59e700 1 -- 172.21.15.139:0/3555948303 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99c059100 con 0x7fd99c071cd0 2021-11-20T05:59:09.718 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.727+0000 7fd992ffd700 1 -- 172.21.15.139:0/3555948303 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd98c016070 con 0x7fd99c071cd0 2021-11-20T05:59:09.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:09 smithi139 conmon[33501]: audit 2021-11-20T05:59:08.599280 2021-11-20T05:59:09.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:09 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 62 : audit [INF] from='client.? 172.21.15.139:0/1937789836' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2021-11-20T05:59:09.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:09 smithi139 conmon[33501]: cluster 2021-11-20T05:59:08.599326+0000 mon.smithi139 (mon.0) 63 : cluster [DBG] mgrmap e8: smithi139.dplyxn(active, since 16s) 2021-11-20T05:59:09.907 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.919+0000 7fd999d9b700 1 -- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fd97c02fdb0 msgr2=0x7fd97c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/1220659339 2021-11-20T05:59:09.907 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.919+0000 7fd999d9b700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fd97c02fdb0 0x7fd97c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T05:59:09.932 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:09.944+0000 7fd99b59e700 1 -- 172.21.15.139:0/3555948303 --> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fd99c106870 con 0x7fd97c02fdb0 2021-11-20T05:59:10.309 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:10.320+0000 7fd999d9b700 1 -- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fd97c02fdb0 msgr2=0x7fd97c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/1220659339 2021-11-20T05:59:10.310 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:10.320+0000 7fd999d9b700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fd97c02fdb0 0x7fd97c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T05:59:11.108 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:11.121+0000 7fd999d9b700 1 -- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fd97c02fdb0 msgr2=0x7fd97c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/1220659339 2021-11-20T05:59:11.109 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:11.121+0000 7fd999d9b700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fd97c02fdb0 0x7fd97c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-20T05:59:12.710 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:12.722+0000 7fd999d9b700 1 -- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fd97c02fdb0 msgr2=0x7fd97c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/1220659339 2021-11-20T05:59:12.710 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:12.722+0000 7fd999d9b700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fd97c02fdb0 0x7fd97c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-20T05:59:13.565 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:13.576+0000 7fd992ffd700 1 -- 172.21.15.139:0/3555948303 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mgrmap(e 9) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fd98c024790 con 0x7fd99c071cd0 2021-11-20T05:59:13.565 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:13.576+0000 7fd992ffd700 1 -- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fd97c02fdb0 msgr2=0x7fd97c032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T05:59:13.566 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:13.576+0000 7fd992ffd700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fd97c02fdb0 0x7fd97c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:59:14.567 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.578+0000 7fd992ffd700 1 -- 172.21.15.139:0/3555948303 <== mon.0 v2:172.21.15.139:3300/0 8 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fd98c010360 con 0x7fd99c071cd0 2021-11-20T05:59:14.567 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.578+0000 7fd992ffd700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7fd97c033c20 0x7fd97c036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:14.567 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.578+0000 7fd992ffd700 1 -- 172.21.15.139:0/3555948303 --> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fd99c106870 con 0x7fd97c033c20 2021-11-20T05:59:14.568 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.579+0000 7fd999d9b700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7fd97c033c20 0x7fd97c036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:14.568 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.579+0000 7fd999d9b700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7fd97c033c20 0x7fd97c036010 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fd9840036d0 tx=0x7fd984006cb0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:14.570 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.583+0000 7fd992ffd700 1 -- 172.21.15.139:0/3555948303 <== mgr.14142 v2:172.21.15.139:6800/3637001875 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fd99c106870 con 0x7fd97c033c20 2021-11-20T05:59:14.571 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: Scheduled prometheus update... 2021-11-20T05:59:14.573 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.585+0000 7fd99b59e700 1 -- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7fd97c033c20 msgr2=0x7fd97c036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:14.573 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.585+0000 7fd99b59e700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7fd97c033c20 0x7fd97c036010 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fd9840036d0 tx=0x7fd984006cb0).stop 2021-11-20T05:59:14.573 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.585+0000 7fd99b59e700 1 -- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd99c071cd0 msgr2=0x7fd99c10c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:14.574 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.585+0000 7fd99b59e700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd99c071cd0 0x7fd99c10c9d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd98c00b1d0 tx=0x7fd98c006ee0).stop 2021-11-20T05:59:14.574 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.585+0000 7fd99b59e700 1 -- 172.21.15.139:0/3555948303 shutdown_connections 2021-11-20T05:59:14.574 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.585+0000 7fd99b59e700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7fd97c033c20 0x7fd97c036010 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:14.574 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.585+0000 7fd99b59e700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd99c071cd0 0x7fd99c10c9d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:14.575 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.585+0000 7fd99b59e700 1 --2- 172.21.15.139:0/3555948303 >> [v2:172.21.15.139:6800/1220659339,v1:172.21.15.139:6801/1220659339] conn(0x7fd97c02fdb0 0x7fd97c032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:59:14.575 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.585+0000 7fd99b59e700 1 -- 172.21.15.139:0/3555948303 >> 172.21.15.139:0/3555948303 conn(0x7fd99c0f71e0 msgr2=0x7fd99c0f8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:14.575 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.585+0000 7fd99b59e700 1 -- 172.21.15.139:0/3555948303 shutdown_connections 2021-11-20T05:59:14.575 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:14.585+0000 7fd99b59e700 1 -- 172.21.15.139:0/3555948303 wait complete. 2021-11-20T05:59:14.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: cluster 2021-11-20T05:59:13.525735+0000 mon.smithi139 (mon.0) 64 : cluster [INF] Active manager daemon smithi139.dplyxn restarted 2021-11-20T05:59:14.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: cluster 2021-11-20T05:59: 2021-11-20T05:59:14.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: 13.526248+0000 mon.smithi139 (mon.0) 65 : cluster [INF] Activating manager daemon smithi139.dplyxn 2021-11-20T05:59:14.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: cluster 2021-11-20T05:59:13.528011+0000 mon.smithi139 (mon.0) 66 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2021-11-20T05:59:14.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: cluster 2021-11-20T05:59:13.579015+0000 mon.smithi139 (mon.0) 67 : cluster [DBG] mgrmap e9: smithi139.dplyxn(active, starting, since 0.0528868s) 2021-11-20T05:59:14.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: audit 2021-11-20T05:59:13.579975+0000 mon.smithi139 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.139:0/3370115087' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi139"}]: dispatch 2021-11-20T05:59:14.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: audit 2021-11-20T05:59:13.580524+0000 mon.smithi139 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.139:0/3370115087' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mgr metadata", "who": "smithi139.dplyxn", "id": "smithi139.dplyxn"}]: dispatch 2021-11-20T05:59:14.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: audit 2021-11-20T05:59:13.581161+0000 mon.smithi139 (mon.0) 70 2021-11-20T05:59:14.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: : audit [DBG] from='mgr.14142 172.21.15.139:0/3370115087' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T05:59:14.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: audit 2021-11-20T05:59:13.581319+0000 mon.smithi139 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.139:0/3370115087' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T05:59:14.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: audit 2021-11-20T05:59:13.581445+0000 mon.smithi139 (mon.0) 72 : audit 2021-11-20T05:59:14.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: [DBG] from='mgr.14142 172.21.15.139:0/3370115087' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T05:59:14.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: cluster 2021-11-20T05:59:14.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: 2021-11-20T05:59:13.595073+0000 mon.smithi139 (mon.0) 73 : cluster 2021-11-20T05:59:14.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: [INF] Manager daemon smithi139.dplyxn is now available 2021-11-20T05:59:14.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: audit 2021-11-20T05:59:14.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: 2021-11-20T05:59:13.694696+0000 mon.smithi139 (mon.0) 2021-11-20T05:59:14.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: 74 : audit [DBG] from='mgr.14142 172.21.15.139:0/3370115087' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:59:14.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: audit 2021-11-20T05:59:13.696166+0000 2021-11-20T05:59:14.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: mon.smithi139 (mon.0) 75 : audit [DBG] from='mgr.14142 172.21.15.139:0/3370115087' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:59:14.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: audit 2021-11-20T05 2021-11-20T05:59:14.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: :59:13.711749+0000 mon.smithi139 (mon.0) 76 : 2021-11-20T05:59:14.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: audit [INF] from='mgr.14142 172.21.15.139:0/3370115087' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:59:14.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: audit 2021-11-20T05:59:14.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: 2021-11-20T05:59:13.719407+0000 mon.smithi139 (mon.0 2021-11-20T05:59:14.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:14 smithi139 conmon[33501]: ) 77 : audit [INF] from='mgr.14142 172.21.15.139:0/3370115087' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T05:59:14.882 INFO:teuthology.orchestra.run.smithi139.stderr:Deploying grafana service with default placement... 2021-11-20T05:59:14.883 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2021-11-20T05:59:15.554 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.565+0000 7f5f65317700 1 Processor -- start 2021-11-20T05:59:15.554 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.565+0000 7f5f65317700 1 -- start start 2021-11-20T05:59:15.554 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.565+0000 7f5f65317700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f600fd2a0 0x7f5f600fd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:15.555 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.565+0000 7f5f65317700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600fdbb0 con 0x7f5f600fd2a0 2021-11-20T05:59:15.555 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.565+0000 7f5f5effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f600fd2a0 0x7f5f600fd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:15.555 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.565+0000 7f5f5effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f600fd2a0 0x7f5f600fd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56640/0 (socket says 172.21.15.139:56640) 2021-11-20T05:59:15.555 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.565+0000 7f5f5effd700 1 -- 172.21.15.139:0/1707123422 learned_addr learned my addr 172.21.15.139:0/1707123422 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:15.556 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.566+0000 7f5f5effd700 1 -- 172.21.15.139:0/1707123422 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f600fdcf0 con 0x7f5f600fd2a0 2021-11-20T05:59:15.556 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.566+0000 7f5f5effd700 1 --2- 172.21.15.139:0/1707123422 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f600fd2a0 0x7f5f600fd6c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f5f5000b5f0 tx=0x7f5f50006ee0).ready entity=mon.0 client_cookie=4a23d21c42e062bd server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:15.556 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.566+0000 7f5f5dffb700 1 -- 172.21.15.139:0/1707123422 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f5000d600 con 0x7f5f600fd2a0 2021-11-20T05:59:15.556 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.566+0000 7f5f5dffb700 1 -- 172.21.15.139:0/1707123422 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f5000dc10 con 0x7f5f600fd2a0 2021-11-20T05:59:15.557 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.566+0000 7f5f5dffb700 1 -- 172.21.15.139:0/1707123422 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f50011cd0 con 0x7f5f600fd2a0 2021-11-20T05:59:15.559 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.567+0000 7f5f65317700 1 -- 172.21.15.139:0/1707123422 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f600fd2a0 msgr2=0x7f5f600fd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:15.559 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.567+0000 7f5f65317700 1 --2- 172.21.15.139:0/1707123422 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f600fd2a0 0x7f5f600fd6c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f5f5000b5f0 tx=0x7f5f50006ee0).stop 2021-11-20T05:59:15.560 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.567+0000 7f5f65317700 1 -- 172.21.15.139:0/1707123422 shutdown_connections 2021-11-20T05:59:15.560 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.567+0000 7f5f65317700 1 --2- 172.21.15.139:0/1707123422 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f600fd2a0 0x7f5f600fd6c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:15.560 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.567+0000 7f5f65317700 1 -- 172.21.15.139:0/1707123422 >> 172.21.15.139:0/1707123422 conn(0x7f5f600f5a80 msgr2=0x7f5f600f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:15.560 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.567+0000 7f5f65317700 1 -- 172.21.15.139:0/1707123422 shutdown_connections 2021-11-20T05:59:15.561 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.567+0000 7f5f65317700 1 -- 172.21.15.139:0/1707123422 wait complete. 2021-11-20T05:59:15.561 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.568+0000 7f5f65317700 1 Processor -- start 2021-11-20T05:59:15.561 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.568+0000 7f5f65317700 1 -- start start 2021-11-20T05:59:15.561 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.568+0000 7f5f65317700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f600fd2a0 0x7f5f600fcb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:15.561 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.568+0000 7f5f65317700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600faff0 con 0x7f5f600fd2a0 2021-11-20T05:59:15.562 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.568+0000 7f5f5effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f600fd2a0 0x7f5f600fcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:15.562 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.568+0000 7f5f5effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f600fd2a0 0x7f5f600fcb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56642/0 (socket says 172.21.15.139:56642) 2021-11-20T05:59:15.562 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.568+0000 7f5f5effd700 1 -- 172.21.15.139:0/1708939647 learned_addr learned my addr 172.21.15.139:0/1708939647 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:15.562 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.569+0000 7f5f5effd700 1 -- 172.21.15.139:0/1708939647 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5000b200 con 0x7f5f600fd2a0 2021-11-20T05:59:15.563 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.569+0000 7f5f5effd700 1 --2- 172.21.15.139:0/1708939647 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f600fd2a0 0x7f5f600fcb70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f5f5000dfb0 tx=0x7f5f50006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:15.563 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.569+0000 7f5f477fe700 1 -- 172.21.15.139:0/1708939647 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f50011df0 con 0x7f5f600fd2a0 2021-11-20T05:59:15.563 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.569+0000 7f5f477fe700 1 -- 172.21.15.139:0/1708939647 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f5001b8c0 con 0x7f5f600fd2a0 2021-11-20T05:59:15.564 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.569+0000 7f5f65317700 1 -- 172.21.15.139:0/1708939647 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f600fb190 con 0x7f5f600fd2a0 2021-11-20T05:59:15.564 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.569+0000 7f5f477fe700 1 -- 172.21.15.139:0/1708939647 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f5001a7c0 con 0x7f5f600fd2a0 2021-11-20T05:59:15.564 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.569+0000 7f5f65317700 1 -- 172.21.15.139:0/1708939647 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f600fb5a0 con 0x7f5f600fd2a0 2021-11-20T05:59:15.564 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.570+0000 7f5f477fe700 1 -- 172.21.15.139:0/1708939647 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f5f5000f070 con 0x7f5f600fd2a0 2021-11-20T05:59:15.565 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.570+0000 7f5f477fe700 1 --2- 172.21.15.139:0/1708939647 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f5f4c02fcf0 0x7f5f4c0321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:15.565 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.570+0000 7f5f477fe700 1 -- 172.21.15.139:0/1708939647 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f5f50040f20 con 0x7f5f600fd2a0 2021-11-20T05:59:15.565 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.570+0000 7f5f65317700 1 -- 172.21.15.139:0/1708939647 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f60045ad0 con 0x7f5f600fd2a0 2021-11-20T05:59:15.565 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.571+0000 7f5f5e7fc700 1 --2- 172.21.15.139:0/1708939647 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f5f4c02fcf0 0x7f5f4c0321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:15.565 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.571+0000 7f5f5e7fc700 1 --2- 172.21.15.139:0/1708939647 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f5f4c02fcf0 0x7f5f4c0321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5f540096a0 tx=0x7f5f54006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:15.566 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.575+0000 7f5f477fe700 1 -- 172.21.15.139:0/1708939647 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f50016070 con 0x7f5f600fd2a0 2021-11-20T05:59:15.756 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.768+0000 7f5f65317700 1 -- 172.21.15.139:0/1708939647 --> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f600008d0 con 0x7f5f4c02fcf0 2021-11-20T05:59:15.760 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.773+0000 7f5f477fe700 1 -- 172.21.15.139:0/1708939647 <== mgr.14142 v2:172.21.15.139:6800/3637001875 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f5f600008d0 con 0x7f5f4c02fcf0 2021-11-20T05:59:15.760 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: Scheduled grafana update... 2021-11-20T05:59:15.762 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.775+0000 7f5f65317700 1 -- 172.21.15.139:0/1708939647 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f5f4c02fcf0 msgr2=0x7f5f4c0321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:15.762 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.775+0000 7f5f65317700 1 --2- 172.21.15.139:0/1708939647 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f5f4c02fcf0 0x7f5f4c0321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5f540096a0 tx=0x7f5f54006b40).stop 2021-11-20T05:59:15.763 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.775+0000 7f5f65317700 1 -- 172.21.15.139:0/1708939647 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f600fd2a0 msgr2=0x7f5f600fcb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:15.763 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.775+0000 7f5f65317700 1 --2- 172.21.15.139:0/1708939647 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f600fd2a0 0x7f5f600fcb70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f5f5000dfb0 tx=0x7f5f50006ee0).stop 2021-11-20T05:59:15.764 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.775+0000 7f5f65317700 1 -- 172.21.15.139:0/1708939647 shutdown_connections 2021-11-20T05:59:15.764 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.775+0000 7f5f65317700 1 --2- 172.21.15.139:0/1708939647 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f5f4c02fcf0 0x7f5f4c0321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:15.764 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.775+0000 7f5f65317700 1 --2- 172.21.15.139:0/1708939647 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f600fd2a0 0x7f5f600fcb70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:15.764 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.775+0000 7f5f65317700 1 -- 172.21.15.139:0/1708939647 >> 172.21.15.139:0/1708939647 conn(0x7f5f600f5a80 msgr2=0x7f5f600ff2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:15.765 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.776+0000 7f5f65317700 1 -- 172.21.15.139:0/1708939647 shutdown_connections 2021-11-20T05:59:15.765 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:15.776+0000 7f5f65317700 1 -- 172.21.15.139:0/1708939647 wait complete. 2021-11-20T05:59:15.832 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:15 smithi139 conmon[33501]: cluster 2021-11-20T05:59:14.581804+0000 mon.smithi139 (mon.0) 78 : cluster [DBG] mgrmap e10: smithi139.dplyxn(active, since 1.05567s) 2021-11-20T05:59:15.833 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:15 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:14.582745+0000 mgr.smithi139.dplyxn (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2021-11-20T05:59:15.833 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:15 smithi139 conmon[33501]: audit 2021-11-20T05:59:14.584776+0000 mon.smithi139 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.139:0/3370115087' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:15.833 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:15 smithi139 conmon[33501]: audit 2021-11-20T05:59:15.407956+0000 mon.smithi139 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.139:0/3370115087' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:16.171 INFO:teuthology.orchestra.run.smithi139.stderr:Deploying node-exporter service with default placement... 2021-11-20T05:59:16.171 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2021-11-20T05:59:16.839 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.846+0000 7f2501533700 1 Processor -- start 2021-11-20T05:59:16.839 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.846+0000 7f2501533700 1 -- start start 2021-11-20T05:59:16.839 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.847+0000 7f2501533700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24fc06fbf0 0x7f24fc0fe870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:16.840 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.847+0000 7f2501533700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc0fed60 con 0x7f24fc06fbf0 2021-11-20T05:59:16.840 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.847+0000 7f24faffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24fc06fbf0 0x7f24fc0fe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:16.840 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.847+0000 7f24faffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24fc06fbf0 0x7f24fc0fe870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56646/0 (socket says 172.21.15.139:56646) 2021-11-20T05:59:16.840 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.847+0000 7f24faffd700 1 -- 172.21.15.139:0/4054719395 learned_addr learned my addr 172.21.15.139:0/4054719395 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:16.841 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.847+0000 7f24faffd700 1 -- 172.21.15.139:0/4054719395 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24fc0feea0 con 0x7f24fc06fbf0 2021-11-20T05:59:16.841 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.847+0000 7f24faffd700 1 --2- 172.21.15.139:0/4054719395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24fc06fbf0 0x7f24fc0fe870 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f24ec00b5f0 tx=0x7f24ec006ee0).ready entity=mon.0 client_cookie=f77dae78daa029fa server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:16.841 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.848+0000 7f24f9ffb700 1 -- 172.21.15.139:0/4054719395 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24ec00d600 con 0x7f24fc06fbf0 2021-11-20T05:59:16.841 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.848+0000 7f24f9ffb700 1 -- 172.21.15.139:0/4054719395 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24ec00dc10 con 0x7f24fc06fbf0 2021-11-20T05:59:16.842 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.848+0000 7f24f9ffb700 1 -- 172.21.15.139:0/4054719395 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24ec011d00 con 0x7f24fc06fbf0 2021-11-20T05:59:16.842 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.848+0000 7f2501533700 1 -- 172.21.15.139:0/4054719395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24fc06fbf0 msgr2=0x7f24fc0fe870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:16.842 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.848+0000 7f2501533700 1 --2- 172.21.15.139:0/4054719395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24fc06fbf0 0x7f24fc0fe870 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f24ec00b5f0 tx=0x7f24ec006ee0).stop 2021-11-20T05:59:16.842 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.848+0000 7f2501533700 1 -- 172.21.15.139:0/4054719395 shutdown_connections 2021-11-20T05:59:16.843 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.848+0000 7f2501533700 1 --2- 172.21.15.139:0/4054719395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24fc06fbf0 0x7f24fc0fe870 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:16.843 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.848+0000 7f2501533700 1 -- 172.21.15.139:0/4054719395 >> 172.21.15.139:0/4054719395 conn(0x7f24fc0f6310 msgr2=0x7f24fc0f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:16.843 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.848+0000 7f2501533700 1 -- 172.21.15.139:0/4054719395 shutdown_connections 2021-11-20T05:59:16.843 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.849+0000 7f2501533700 1 -- 172.21.15.139:0/4054719395 wait complete. 2021-11-20T05:59:16.843 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.849+0000 7f2501533700 1 Processor -- start 2021-11-20T05:59:16.844 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.849+0000 7f2501533700 1 -- start start 2021-11-20T05:59:16.844 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.849+0000 7f2501533700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24fc06fbf0 0x7f24fc10aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:16.844 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.849+0000 7f2501533700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc10af40 con 0x7f24fc06fbf0 2021-11-20T05:59:16.844 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.849+0000 7f24faffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24fc06fbf0 0x7f24fc10aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:16.845 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.849+0000 7f24faffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24fc06fbf0 0x7f24fc10aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56648/0 (socket says 172.21.15.139:56648) 2021-11-20T05:59:16.845 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.849+0000 7f24faffd700 1 -- 172.21.15.139:0/3297129073 learned_addr learned my addr 172.21.15.139:0/3297129073 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:16.845 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.850+0000 7f24faffd700 1 -- 172.21.15.139:0/3297129073 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec00b200 con 0x7f24fc06fbf0 2021-11-20T05:59:16.845 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.850+0000 7f24faffd700 1 --2- 172.21.15.139:0/3297129073 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24fc06fbf0 0x7f24fc10aa50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f24ec00b1d0 tx=0x7f24ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:16.846 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.850+0000 7f24e37fe700 1 -- 172.21.15.139:0/3297129073 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24ec011e20 con 0x7f24fc06fbf0 2021-11-20T05:59:16.846 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.850+0000 7f24e37fe700 1 -- 172.21.15.139:0/3297129073 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24ec01b8c0 con 0x7f24fc06fbf0 2021-11-20T05:59:16.846 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.850+0000 7f24e37fe700 1 -- 172.21.15.139:0/3297129073 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24ec01a7d0 con 0x7f24fc06fbf0 2021-11-20T05:59:16.847 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.850+0000 7f2501533700 1 -- 172.21.15.139:0/3297129073 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24fc10b140 con 0x7f24fc06fbf0 2021-11-20T05:59:16.847 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.850+0000 7f2501533700 1 -- 172.21.15.139:0/3297129073 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24fc10b550 con 0x7f24fc06fbf0 2021-11-20T05:59:16.847 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.851+0000 7f24e37fe700 1 -- 172.21.15.139:0/3297129073 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f24ec00f070 con 0x7f24fc06fbf0 2021-11-20T05:59:16.847 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.851+0000 7f24e37fe700 1 --2- 172.21.15.139:0/3297129073 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f24e802feb0 0x7f24e8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:16.848 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.851+0000 7f24e37fe700 1 -- 172.21.15.139:0/3297129073 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f24ec040f60 con 0x7f24fc06fbf0 2021-11-20T05:59:16.848 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.851+0000 7f2501533700 1 -- 172.21.15.139:0/3297129073 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24fc045ad0 con 0x7f24fc06fbf0 2021-11-20T05:59:16.848 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.851+0000 7f24fa7fc700 1 --2- 172.21.15.139:0/3297129073 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f24e802feb0 0x7f24e8032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:16.848 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.852+0000 7f24fa7fc700 1 --2- 172.21.15.139:0/3297129073 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f24e802feb0 0x7f24e8032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f24f00096a0 tx=0x7f24f0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:16.849 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:16.855+0000 7f24e37fe700 1 -- 172.21.15.139:0/3297129073 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f24ec024410 con 0x7f24fc06fbf0 2021-11-20T05:59:17.019 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:16 smithi139 conmon[33501]: audit 2021-11-20T05:59:15.771288+0000 mgr.smithi139.dplyxn (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:59:17.020 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:16 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:15.771802+0000 mgr.smithi139.dplyxn (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2021-11-20T05:59:17.020 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:16 smithi139 conmon[33501]: audit 2021-11-20T05:59:15.774780+0000 mon.smithi139 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.139:0/3370115087' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:17.020 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:16 smithi139 conmon[33501]: cluster 2021-11-20T05:59:16.409575+0000 mon.smithi139 (mon.0) 82 : cluster [DBG] mgrmap e11: smithi139.dplyxn(active, since 2s) 2021-11-20T05:59:17.021 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.031+0000 7f2501533700 1 -- 172.21.15.139:0/3297129073 --> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f24fc104950 con 0x7f24e802feb0 2021-11-20T05:59:17.023 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.036+0000 7f24e37fe700 1 -- 172.21.15.139:0/3297129073 <== mgr.14142 v2:172.21.15.139:6800/3637001875 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f24fc104950 con 0x7f24e802feb0 2021-11-20T05:59:17.023 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2021-11-20T05:59:17.026 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.037+0000 7f2501533700 1 -- 172.21.15.139:0/3297129073 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f24e802feb0 msgr2=0x7f24e8032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:17.026 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.037+0000 7f2501533700 1 --2- 172.21.15.139:0/3297129073 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f24e802feb0 0x7f24e8032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f24f00096a0 tx=0x7f24f0006b40).stop 2021-11-20T05:59:17.026 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.037+0000 7f2501533700 1 -- 172.21.15.139:0/3297129073 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24fc06fbf0 msgr2=0x7f24fc10aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:17.027 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.037+0000 7f2501533700 1 --2- 172.21.15.139:0/3297129073 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24fc06fbf0 0x7f24fc10aa50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f24ec00b1d0 tx=0x7f24ec006ee0).stop 2021-11-20T05:59:17.027 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.037+0000 7f2501533700 1 -- 172.21.15.139:0/3297129073 shutdown_connections 2021-11-20T05:59:17.027 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.037+0000 7f2501533700 1 --2- 172.21.15.139:0/3297129073 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f24e802feb0 0x7f24e8032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:17.027 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.037+0000 7f2501533700 1 --2- 172.21.15.139:0/3297129073 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f24fc06fbf0 0x7f24fc10aa50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:17.027 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.037+0000 7f2501533700 1 -- 172.21.15.139:0/3297129073 >> 172.21.15.139:0/3297129073 conn(0x7f24fc0f6310 msgr2=0x7f24fc0f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:17.028 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.038+0000 7f2501533700 1 -- 172.21.15.139:0/3297129073 shutdown_connections 2021-11-20T05:59:17.028 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.038+0000 7f2501533700 1 -- 172.21.15.139:0/3297129073 wait complete. 2021-11-20T05:59:17.406 INFO:teuthology.orchestra.run.smithi139.stderr:Deploying alertmanager service with default placement... 2021-11-20T05:59:17.406 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2021-11-20T05:59:17.850 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.859+0000 7f566f7e4700 1 Processor -- start 2021-11-20T05:59:17.850 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.859+0000 7f566f7e4700 1 -- start start 2021-11-20T05:59:17.851 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.859+0000 7f566f7e4700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56680faae0 0x7f56680faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:17.851 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.860+0000 7f566f7e4700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56680fb480 con 0x7f56680faae0 2021-11-20T05:59:17.851 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.860+0000 7f566d580700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56680faae0 0x7f56680faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:17.851 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.860+0000 7f566d580700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56680faae0 0x7f56680faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56652/0 (socket says 172.21.15.139:56652) 2021-11-20T05:59:17.852 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.860+0000 7f566d580700 1 -- 172.21.15.139:0/1461638214 learned_addr learned my addr 172.21.15.139:0/1461638214 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:17.852 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.860+0000 7f566d580700 1 -- 172.21.15.139:0/1461638214 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56680fb5c0 con 0x7f56680faae0 2021-11-20T05:59:17.852 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.860+0000 7f566d580700 1 --2- 172.21.15.139:0/1461638214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56680faae0 0x7f56680faf00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5664009960 tx=0x7f56640092d0).ready entity=mon.0 client_cookie=7a27908c8e6973a8 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:17.852 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.860+0000 7f5657fff700 1 -- 172.21.15.139:0/1461638214 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f566400b9c0 con 0x7f56680faae0 2021-11-20T05:59:17.852 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.860+0000 7f5657fff700 1 -- 172.21.15.139:0/1461638214 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5664011400 con 0x7f56680faae0 2021-11-20T05:59:17.853 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.860+0000 7f5657fff700 1 -- 172.21.15.139:0/1461638214 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5664010580 con 0x7f56680faae0 2021-11-20T05:59:17.853 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.861+0000 7f566f7e4700 1 -- 172.21.15.139:0/1461638214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56680faae0 msgr2=0x7f56680faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:17.853 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.861+0000 7f566f7e4700 1 --2- 172.21.15.139:0/1461638214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56680faae0 0x7f56680faf00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5664009960 tx=0x7f56640092d0).stop 2021-11-20T05:59:17.853 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.861+0000 7f566f7e4700 1 -- 172.21.15.139:0/1461638214 shutdown_connections 2021-11-20T05:59:17.853 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.861+0000 7f566f7e4700 1 --2- 172.21.15.139:0/1461638214 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56680faae0 0x7f56680faf00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:17.854 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.861+0000 7f566f7e4700 1 -- 172.21.15.139:0/1461638214 >> 172.21.15.139:0/1461638214 conn(0x7f56680f6280 msgr2=0x7f56680f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:17.854 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.861+0000 7f566f7e4700 1 -- 172.21.15.139:0/1461638214 shutdown_connections 2021-11-20T05:59:17.854 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.861+0000 7f566f7e4700 1 -- 172.21.15.139:0/1461638214 wait complete. 2021-11-20T05:59:17.854 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.862+0000 7f566f7e4700 1 Processor -- start 2021-11-20T05:59:17.855 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.862+0000 7f566f7e4700 1 -- start start 2021-11-20T05:59:17.855 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.862+0000 7f566f7e4700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56680faae0 0x7f566810fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:17.855 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.862+0000 7f566f7e4700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566810b560 con 0x7f56680faae0 2021-11-20T05:59:17.855 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.862+0000 7f566d580700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56680faae0 0x7f566810fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:17.856 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.862+0000 7f566d580700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56680faae0 0x7f566810fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56654/0 (socket says 172.21.15.139:56654) 2021-11-20T05:59:17.856 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.862+0000 7f566d580700 1 -- 172.21.15.139:0/238010522 learned_addr learned my addr 172.21.15.139:0/238010522 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:17.856 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.862+0000 7f566d580700 1 -- 172.21.15.139:0/238010522 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5664009570 con 0x7f56680faae0 2021-11-20T05:59:17.856 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.862+0000 7f566d580700 1 --2- 172.21.15.139:0/238010522 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56680faae0 0x7f566810fb70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f5664006f00 tx=0x7f56640092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:17.856 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.862+0000 7f5655ffb700 1 -- 172.21.15.139:0/238010522 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56640103a0 con 0x7f56680faae0 2021-11-20T05:59:17.857 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.862+0000 7f566f7e4700 1 -- 172.21.15.139:0/238010522 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566806fc50 con 0x7f56680faae0 2021-11-20T05:59:17.857 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.862+0000 7f5655ffb700 1 -- 172.21.15.139:0/238010522 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5664010cf0 con 0x7f56680faae0 2021-11-20T05:59:17.857 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.862+0000 7f5655ffb700 1 -- 172.21.15.139:0/238010522 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f566401bab0 con 0x7f56680faae0 2021-11-20T05:59:17.857 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.863+0000 7f566f7e4700 1 -- 172.21.15.139:0/238010522 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5668110060 con 0x7f56680faae0 2021-11-20T05:59:17.858 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.863+0000 7f5655ffb700 1 -- 172.21.15.139:0/238010522 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f566400f070 con 0x7f56680faae0 2021-11-20T05:59:17.859 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.864+0000 7f5655ffb700 1 --2- 172.21.15.139:0/238010522 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f565002fe60 0x7f5650032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:17.859 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.864+0000 7f566f7e4700 1 -- 172.21.15.139:0/238010522 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5668045ad0 con 0x7f56680faae0 2021-11-20T05:59:17.859 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.864+0000 7f5655ffb700 1 -- 172.21.15.139:0/238010522 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f5664041ac0 con 0x7f56680faae0 2021-11-20T05:59:17.859 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.864+0000 7f566cd7f700 1 --2- 172.21.15.139:0/238010522 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f565002fe60 0x7f5650032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:17.860 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.864+0000 7f566cd7f700 1 --2- 172.21.15.139:0/238010522 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f565002fe60 0x7f5650032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f56580096a0 tx=0x7f5658006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:17.860 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:17.868+0000 7f5655ffb700 1 -- 172.21.15.139:0/238010522 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5664024cb0 con 0x7f56680faae0 2021-11-20T05:59:18.032 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.045+0000 7f566f7e4700 1 -- 172.21.15.139:0/238010522 --> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f5668057f10 con 0x7f565002fe60 2021-11-20T05:59:18.035 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.048+0000 7f5655ffb700 1 -- 172.21.15.139:0/238010522 <== mgr.14142 v2:172.21.15.139:6800/3637001875 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f5668057f10 con 0x7f565002fe60 2021-11-20T05:59:18.036 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2021-11-20T05:59:18.039 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.050+0000 7f566f7e4700 1 -- 172.21.15.139:0/238010522 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f565002fe60 msgr2=0x7f5650032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:18.040 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.050+0000 7f566f7e4700 1 --2- 172.21.15.139:0/238010522 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f565002fe60 0x7f5650032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f56580096a0 tx=0x7f5658006b40).stop 2021-11-20T05:59:18.040 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.050+0000 7f566f7e4700 1 -- 172.21.15.139:0/238010522 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56680faae0 msgr2=0x7f566810fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:18.040 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.050+0000 7f566f7e4700 1 --2- 172.21.15.139:0/238010522 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56680faae0 0x7f566810fb70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f5664006f00 tx=0x7f56640092d0).stop 2021-11-20T05:59:18.041 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.050+0000 7f566f7e4700 1 -- 172.21.15.139:0/238010522 shutdown_connections 2021-11-20T05:59:18.041 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.050+0000 7f566f7e4700 1 --2- 172.21.15.139:0/238010522 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f565002fe60 0x7f5650032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:18.041 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.050+0000 7f566f7e4700 1 --2- 172.21.15.139:0/238010522 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56680faae0 0x7f566810fb70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:18.042 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.050+0000 7f566f7e4700 1 -- 172.21.15.139:0/238010522 >> 172.21.15.139:0/238010522 conn(0x7f56680f6280 msgr2=0x7f56680f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:18.042 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.050+0000 7f566f7e4700 1 -- 172.21.15.139:0/238010522 shutdown_connections 2021-11-20T05:59:18.042 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.050+0000 7f566f7e4700 1 -- 172.21.15.139:0/238010522 wait complete. 2021-11-20T05:59:18.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:18 smithi139 conmon[33501]: audit 2021-11-20T05:59:17.034249+0000 mgr.smithi139.dplyxn (mgr.14142) 5 : audit [DBG] 2021-11-20T05:59:18.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:18 smithi139 conmon[33501]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:59:18.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:18 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:17.035105+0000 mgr.smithi139.dplyxn (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2021-11-20T05:59:18.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:18 smithi139 conmon[33501]: audit 2021-11-20T05:59 2021-11-20T05:59:18.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:18 smithi139 conmon[33501]: :17.037405+0000 mon.smithi139 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.139:0/3370115087' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:18.441 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2021-11-20T05:59:18.907 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.907+0000 7f2b6e28f700 1 Processor -- start 2021-11-20T05:59:18.908 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.907+0000 7f2b6e28f700 1 -- start start 2021-11-20T05:59:18.908 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.907+0000 7f2b6e28f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6806fc30 0x7f2b680fe830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:18.909 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.907+0000 7f2b6e28f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680fed20 con 0x7f2b6806fc30 2021-11-20T05:59:18.909 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.907+0000 7f2b67fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6806fc30 0x7f2b680fe830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:18.910 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.907+0000 7f2b67fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6806fc30 0x7f2b680fe830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56658/0 (socket says 172.21.15.139:56658) 2021-11-20T05:59:18.910 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.907+0000 7f2b67fff700 1 -- 172.21.15.139:0/398296130 learned_addr learned my addr 172.21.15.139:0/398296130 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:18.910 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.908+0000 7f2b67fff700 1 -- 172.21.15.139:0/398296130 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b680fee60 con 0x7f2b6806fc30 2021-11-20T05:59:18.911 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.908+0000 7f2b67fff700 1 --2- 172.21.15.139:0/398296130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6806fc30 0x7f2b680fe830 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f2b5800b5f0 tx=0x7f2b58006ee0).ready entity=mon.0 client_cookie=6052ef41486916d6 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:18.912 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.908+0000 7f2b677fe700 1 -- 172.21.15.139:0/398296130 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b5800d600 con 0x7f2b6806fc30 2021-11-20T05:59:18.912 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.908+0000 7f2b677fe700 1 -- 172.21.15.139:0/398296130 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b5800dc10 con 0x7f2b6806fc30 2021-11-20T05:59:18.913 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.908+0000 7f2b677fe700 1 -- 172.21.15.139:0/398296130 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b58011cd0 con 0x7f2b6806fc30 2021-11-20T05:59:18.913 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.909+0000 7f2b6e28f700 1 -- 172.21.15.139:0/398296130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6806fc30 msgr2=0x7f2b680fe830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:18.914 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.909+0000 7f2b6e28f700 1 --2- 172.21.15.139:0/398296130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6806fc30 0x7f2b680fe830 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f2b5800b5f0 tx=0x7f2b58006ee0).stop 2021-11-20T05:59:18.914 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.909+0000 7f2b6e28f700 1 -- 172.21.15.139:0/398296130 shutdown_connections 2021-11-20T05:59:18.914 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.909+0000 7f2b6e28f700 1 --2- 172.21.15.139:0/398296130 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6806fc30 0x7f2b680fe830 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:18.915 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.909+0000 7f2b6e28f700 1 -- 172.21.15.139:0/398296130 >> 172.21.15.139:0/398296130 conn(0x7f2b680f6310 msgr2=0x7f2b680f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:18.915 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.909+0000 7f2b6e28f700 1 -- 172.21.15.139:0/398296130 shutdown_connections 2021-11-20T05:59:18.916 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.909+0000 7f2b6e28f700 1 -- 172.21.15.139:0/398296130 wait complete. 2021-11-20T05:59:18.916 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.910+0000 7f2b6e28f700 1 Processor -- start 2021-11-20T05:59:18.917 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.910+0000 7f2b6e28f700 1 -- start start 2021-11-20T05:59:18.917 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.910+0000 7f2b6e28f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6806fc30 0x7f2b680fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:18.918 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.910+0000 7f2b6e28f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680fa730 con 0x7f2b6806fc30 2021-11-20T05:59:18.918 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.910+0000 7f2b67fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6806fc30 0x7f2b680fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:18.918 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.910+0000 7f2b67fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6806fc30 0x7f2b680fc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56660/0 (socket says 172.21.15.139:56660) 2021-11-20T05:59:18.919 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.910+0000 7f2b67fff700 1 -- 172.21.15.139:0/2370935569 learned_addr learned my addr 172.21.15.139:0/2370935569 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:18.919 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.911+0000 7f2b67fff700 1 -- 172.21.15.139:0/2370935569 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5800b200 con 0x7f2b6806fc30 2021-11-20T05:59:18.920 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.911+0000 7f2b67fff700 1 --2- 172.21.15.139:0/2370935569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6806fc30 0x7f2b680fc2b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f2b5800dfb0 tx=0x7f2b58006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:18.920 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.911+0000 7f2b657fa700 1 -- 172.21.15.139:0/2370935569 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b5801b410 con 0x7f2b6806fc30 2021-11-20T05:59:18.921 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.911+0000 7f2b657fa700 1 -- 172.21.15.139:0/2370935569 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b5801ba20 con 0x7f2b6806fc30 2021-11-20T05:59:18.921 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.911+0000 7f2b657fa700 1 -- 172.21.15.139:0/2370935569 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b5801a970 con 0x7f2b6806fc30 2021-11-20T05:59:18.922 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.911+0000 7f2b6e28f700 1 -- 172.21.15.139:0/2370935569 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b680fa8d0 con 0x7f2b6806fc30 2021-11-20T05:59:18.923 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.911+0000 7f2b6e28f700 1 -- 172.21.15.139:0/2370935569 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b680face0 con 0x7f2b6806fc30 2021-11-20T05:59:18.923 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.912+0000 7f2b6e28f700 1 -- 172.21.15.139:0/2370935569 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b68045ad0 con 0x7f2b6806fc30 2021-11-20T05:59:18.924 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.912+0000 7f2b657fa700 1 -- 172.21.15.139:0/2370935569 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f2b5800f070 con 0x7f2b6806fc30 2021-11-20T05:59:18.924 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.913+0000 7f2b657fa700 1 --2- 172.21.15.139:0/2370935569 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f2b4c02feb0 0x7f2b4c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:18.924 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.913+0000 7f2b657fa700 1 -- 172.21.15.139:0/2370935569 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f2b580417b0 con 0x7f2b6806fc30 2021-11-20T05:59:18.925 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.913+0000 7f2b5f7fe700 1 --2- 172.21.15.139:0/2370935569 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f2b4c02feb0 0x7f2b4c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:18.925 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.913+0000 7f2b5f7fe700 1 --2- 172.21.15.139:0/2370935569 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f2b4c02feb0 0x7f2b4c032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f2b500096a0 tx=0x7f2b50006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:18.926 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:18.916+0000 7f2b657fa700 1 -- 172.21.15.139:0/2370935569 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b58022b00 con 0x7f2b6806fc30 2021-11-20T05:59:19.052 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.065+0000 7f2b6e28f700 1 -- 172.21.15.139:0/2370935569 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f2b680ffb40 con 0x7f2b6806fc30 2021-11-20T05:59:19.058 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.071+0000 7f2b657fa700 1 -- 172.21.15.139:0/2370935569 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f2b58016070 con 0x7f2b6806fc30 2021-11-20T05:59:19.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.073+0000 7f2b6e28f700 1 -- 172.21.15.139:0/2370935569 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f2b4c02feb0 msgr2=0x7f2b4c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:19.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.073+0000 7f2b6e28f700 1 --2- 172.21.15.139:0/2370935569 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f2b4c02feb0 0x7f2b4c032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f2b500096a0 tx=0x7f2b50006b40).stop 2021-11-20T05:59:19.063 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.073+0000 7f2b6e28f700 1 -- 172.21.15.139:0/2370935569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6806fc30 msgr2=0x7f2b680fc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:19.064 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.073+0000 7f2b6e28f700 1 --2- 172.21.15.139:0/2370935569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6806fc30 0x7f2b680fc2b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f2b5800dfb0 tx=0x7f2b58006ee0).stop 2021-11-20T05:59:19.064 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.073+0000 7f2b6e28f700 1 -- 172.21.15.139:0/2370935569 shutdown_connections 2021-11-20T05:59:19.064 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.073+0000 7f2b6e28f700 1 --2- 172.21.15.139:0/2370935569 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f2b4c02feb0 0x7f2b4c032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:19.064 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.073+0000 7f2b6e28f700 1 --2- 172.21.15.139:0/2370935569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2b6806fc30 0x7f2b680fc2b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:19.064 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.073+0000 7f2b6e28f700 1 -- 172.21.15.139:0/2370935569 >> 172.21.15.139:0/2370935569 conn(0x7f2b680f6310 msgr2=0x7f2b680f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:19.065 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.073+0000 7f2b6e28f700 1 -- 172.21.15.139:0/2370935569 shutdown_connections 2021-11-20T05:59:19.066 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.073+0000 7f2b6e28f700 1 -- 172.21.15.139:0/2370935569 wait complete. 2021-11-20T05:59:19.210 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:19 smithi139 conmon[33501]: audit 2021-11-20T05:59:18.047453+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T05:59:19.210 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:19 smithi139 conmon[33501]: 14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:59:19.211 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:19 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:18.047951+0000 mgr.smithi139.dplyxn (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2021-11-20T05:59:19.211 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:19 smithi139 conmon[33501]: audit 2021-11-20T05:59:18.049872+0000 mon.smithi139 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.139:0/3370115087' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:19.476 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2021-11-20T05:59:19.932 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.939+0000 7ff54d9d2700 1 Processor -- start 2021-11-20T05:59:19.932 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.939+0000 7ff54d9d2700 1 -- start start 2021-11-20T05:59:19.933 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.939+0000 7ff54d9d2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5480faad0 0x7ff5480faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:19.933 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.939+0000 7ff54d9d2700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5480fb3e0 con 0x7ff5480faad0 2021-11-20T05:59:19.933 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.939+0000 7ff546ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5480faad0 0x7ff5480faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:19.933 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.940+0000 7ff546ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5480faad0 0x7ff5480faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56664/0 (socket says 172.21.15.139:56664) 2021-11-20T05:59:19.934 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.940+0000 7ff546ffd700 1 -- 172.21.15.139:0/926910221 learned_addr learned my addr 172.21.15.139:0/926910221 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:19.934 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.940+0000 7ff546ffd700 1 -- 172.21.15.139:0/926910221 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5480fb520 con 0x7ff5480faad0 2021-11-20T05:59:19.934 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.940+0000 7ff546ffd700 1 --2- 172.21.15.139:0/926910221 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5480faad0 0x7ff5480faef0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7ff53800b5f0 tx=0x7ff538006ee0).ready entity=mon.0 client_cookie=288f27fafad7ac75 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:19.934 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.940+0000 7ff5467fc700 1 -- 172.21.15.139:0/926910221 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff53800d600 con 0x7ff5480faad0 2021-11-20T05:59:19.934 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.940+0000 7ff5467fc700 1 -- 172.21.15.139:0/926910221 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff53800dc10 con 0x7ff5480faad0 2021-11-20T05:59:19.935 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.940+0000 7ff5467fc700 1 -- 172.21.15.139:0/926910221 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff538011cd0 con 0x7ff5480faad0 2021-11-20T05:59:19.935 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.941+0000 7ff54d9d2700 1 -- 172.21.15.139:0/926910221 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5480faad0 msgr2=0x7ff5480faef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:19.935 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.941+0000 7ff54d9d2700 1 --2- 172.21.15.139:0/926910221 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5480faad0 0x7ff5480faef0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7ff53800b5f0 tx=0x7ff538006ee0).stop 2021-11-20T05:59:19.935 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.941+0000 7ff54d9d2700 1 -- 172.21.15.139:0/926910221 shutdown_connections 2021-11-20T05:59:19.936 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.941+0000 7ff54d9d2700 1 --2- 172.21.15.139:0/926910221 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5480faad0 0x7ff5480faef0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:19.936 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.941+0000 7ff54d9d2700 1 -- 172.21.15.139:0/926910221 >> 172.21.15.139:0/926910221 conn(0x7ff5480f6310 msgr2=0x7ff5480f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:19.936 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.941+0000 7ff54d9d2700 1 -- 172.21.15.139:0/926910221 shutdown_connections 2021-11-20T05:59:19.936 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.941+0000 7ff54d9d2700 1 -- 172.21.15.139:0/926910221 wait complete. 2021-11-20T05:59:19.937 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.941+0000 7ff54d9d2700 1 Processor -- start 2021-11-20T05:59:19.937 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.942+0000 7ff54d9d2700 1 -- start start 2021-11-20T05:59:19.937 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.942+0000 7ff54d9d2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5480faad0 0x7ff54810cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:19.937 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.942+0000 7ff54d9d2700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54810d0c0 con 0x7ff5480faad0 2021-11-20T05:59:19.937 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.942+0000 7ff546ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5480faad0 0x7ff54810cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:19.938 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.942+0000 7ff546ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5480faad0 0x7ff54810cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56666/0 (socket says 172.21.15.139:56666) 2021-11-20T05:59:19.938 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.942+0000 7ff546ffd700 1 -- 172.21.15.139:0/824905504 learned_addr learned my addr 172.21.15.139:0/824905504 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:19.938 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.942+0000 7ff546ffd700 1 -- 172.21.15.139:0/824905504 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53800b200 con 0x7ff5480faad0 2021-11-20T05:59:19.938 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.942+0000 7ff546ffd700 1 --2- 172.21.15.139:0/824905504 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5480faad0 0x7ff54810cbd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7ff53800dfb0 tx=0x7ff538006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.942+0000 7ff53ffff700 1 -- 172.21.15.139:0/824905504 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff538011df0 con 0x7ff5480faad0 2021-11-20T05:59:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.942+0000 7ff53ffff700 1 -- 172.21.15.139:0/824905504 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff53801b8c0 con 0x7ff5480faad0 2021-11-20T05:59:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.942+0000 7ff54d9d2700 1 -- 172.21.15.139:0/824905504 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54810d2c0 con 0x7ff5480faad0 2021-11-20T05:59:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.942+0000 7ff53ffff700 1 -- 172.21.15.139:0/824905504 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff53801a8c0 con 0x7ff5480faad0 2021-11-20T05:59:19.939 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.942+0000 7ff54d9d2700 1 -- 172.21.15.139:0/824905504 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54810d6d0 con 0x7ff5480faad0 2021-11-20T05:59:19.940 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.943+0000 7ff53ffff700 1 -- 172.21.15.139:0/824905504 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7ff53800f070 con 0x7ff5480faad0 2021-11-20T05:59:19.940 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.943+0000 7ff53ffff700 1 --2- 172.21.15.139:0/824905504 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7ff52802fe60 0x7ff528032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:19.940 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.943+0000 7ff53ffff700 1 -- 172.21.15.139:0/824905504 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ff538041fb0 con 0x7ff5480faad0 2021-11-20T05:59:19.940 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.943+0000 7ff53e7fc700 1 --2- 172.21.15.139:0/824905504 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7ff52802fe60 0x7ff528032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:19.941 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.944+0000 7ff54d9d2700 1 -- 172.21.15.139:0/824905504 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff548059100 con 0x7ff5480faad0 2021-11-20T05:59:19.941 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.944+0000 7ff53e7fc700 1 --2- 172.21.15.139:0/824905504 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7ff52802fe60 0x7ff528032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7ff5300096a0 tx=0x7ff530006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:19.941 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:19.948+0000 7ff53ffff700 1 -- 172.21.15.139:0/824905504 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff538024410 con 0x7ff5480faad0 2021-11-20T05:59:20.087 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.099+0000 7ff54d9d2700 1 -- 172.21.15.139:0/824905504 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7ff548106c00 con 0x7ff5480faad0 2021-11-20T05:59:20.092 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.105+0000 7ff53ffff700 1 -- 172.21.15.139:0/824905504 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7ff538006a70 con 0x7ff5480faad0 2021-11-20T05:59:20.096 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.106+0000 7ff54d9d2700 1 -- 172.21.15.139:0/824905504 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7ff52802fe60 msgr2=0x7ff528032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:20.096 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.106+0000 7ff54d9d2700 1 --2- 172.21.15.139:0/824905504 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7ff52802fe60 0x7ff528032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7ff5300096a0 tx=0x7ff530006b40).stop 2021-11-20T05:59:20.097 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.107+0000 7ff54d9d2700 1 -- 172.21.15.139:0/824905504 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5480faad0 msgr2=0x7ff54810cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:20.097 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.107+0000 7ff54d9d2700 1 --2- 172.21.15.139:0/824905504 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5480faad0 0x7ff54810cbd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7ff53800dfb0 tx=0x7ff538006ee0).stop 2021-11-20T05:59:20.099 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.107+0000 7ff54d9d2700 1 -- 172.21.15.139:0/824905504 shutdown_connections 2021-11-20T05:59:20.101 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.107+0000 7ff54d9d2700 1 --2- 172.21.15.139:0/824905504 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7ff52802fe60 0x7ff528032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:20.102 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.107+0000 7ff54d9d2700 1 --2- 172.21.15.139:0/824905504 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff5480faad0 0x7ff54810cbd0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:20.102 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.107+0000 7ff54d9d2700 1 -- 172.21.15.139:0/824905504 >> 172.21.15.139:0/824905504 conn(0x7ff5480f6310 msgr2=0x7ff5480ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:20.103 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.107+0000 7ff54d9d2700 1 -- 172.21.15.139:0/824905504 shutdown_connections 2021-11-20T05:59:20.103 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.107+0000 7ff54d9d2700 1 -- 172.21.15.139:0/824905504 wait complete. 2021-11-20T05:59:20.251 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:20 smithi139 conmon[33501]: audit 2021-11-20T05:59:19.072559+0000 mon.smithi139 (mon.0) 2021-11-20T05:59:20.251 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:20 smithi139 conmon[33501]: 85 : audit [INF] from='client.? 172.21.15.139:0/2370935569' entity='client.admin' 2021-11-20T05:59:20.411 INFO:teuthology.orchestra.run.smithi139.stderr:Enabling the dashboard module... 2021-11-20T05:59:20.411 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2021-11-20T05:59:20.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.872+0000 7fbcfeeff700 1 Processor -- start 2021-11-20T05:59:20.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.872+0000 7fbcfeeff700 1 -- start start 2021-11-20T05:59:20.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.872+0000 7fbcfeeff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcf806fbf0 0x7fbcf8100a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:20.863 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.872+0000 7fbcfeeff700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf80fc480 con 0x7fbcf806fbf0 2021-11-20T05:59:20.864 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.872+0000 7fbcfdefd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcf806fbf0 0x7fbcf8100a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:20.864 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.872+0000 7fbcfdefd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcf806fbf0 0x7fbcf8100a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56670/0 (socket says 172.21.15.139:56670) 2021-11-20T05:59:20.864 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.872+0000 7fbcfdefd700 1 -- 172.21.15.139:0/4150579496 learned_addr learned my addr 172.21.15.139:0/4150579496 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:20.864 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.872+0000 7fbcfdefd700 1 -- 172.21.15.139:0/4150579496 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf8100f80 con 0x7fbcf806fbf0 2021-11-20T05:59:20.865 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.873+0000 7fbcfdefd700 1 --2- 172.21.15.139:0/4150579496 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcf806fbf0 0x7fbcf8100a90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fbcf400b5f0 tx=0x7fbcf4006ee0).ready entity=mon.0 client_cookie=5abe96b9c23a2372 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:20.865 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.873+0000 7fbcfcefb700 1 -- 172.21.15.139:0/4150579496 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcf400d600 con 0x7fbcf806fbf0 2021-11-20T05:59:20.865 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.873+0000 7fbcfcefb700 1 -- 172.21.15.139:0/4150579496 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcf400dc10 con 0x7fbcf806fbf0 2021-11-20T05:59:20.865 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.873+0000 7fbcfcefb700 1 -- 172.21.15.139:0/4150579496 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcf4011d00 con 0x7fbcf806fbf0 2021-11-20T05:59:20.866 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.873+0000 7fbcfeeff700 1 -- 172.21.15.139:0/4150579496 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcf806fbf0 msgr2=0x7fbcf8100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:20.866 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.873+0000 7fbcfeeff700 1 --2- 172.21.15.139:0/4150579496 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcf806fbf0 0x7fbcf8100a90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fbcf400b5f0 tx=0x7fbcf4006ee0).stop 2021-11-20T05:59:20.866 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.873+0000 7fbcfeeff700 1 -- 172.21.15.139:0/4150579496 shutdown_connections 2021-11-20T05:59:20.866 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.873+0000 7fbcfeeff700 1 --2- 172.21.15.139:0/4150579496 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcf806fbf0 0x7fbcf8100a90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:20.867 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.873+0000 7fbcfeeff700 1 -- 172.21.15.139:0/4150579496 >> 172.21.15.139:0/4150579496 conn(0x7fbcf80f6110 msgr2=0x7fbcf80f8530 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:20.867 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.873+0000 7fbcfeeff700 1 -- 172.21.15.139:0/4150579496 shutdown_connections 2021-11-20T05:59:20.867 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.874+0000 7fbcfeeff700 1 -- 172.21.15.139:0/4150579496 wait complete. 2021-11-20T05:59:20.867 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.874+0000 7fbcfeeff700 1 Processor -- start 2021-11-20T05:59:20.867 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.874+0000 7fbcfeeff700 1 -- start start 2021-11-20T05:59:20.868 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.874+0000 7fbcfeeff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcf806fbf0 0x7fbcf810ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:20.868 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.874+0000 7fbcfeeff700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf810cef0 con 0x7fbcf806fbf0 2021-11-20T05:59:20.868 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.874+0000 7fbcfdefd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcf806fbf0 0x7fbcf810ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:20.868 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.874+0000 7fbcfdefd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcf806fbf0 0x7fbcf810ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56672/0 (socket says 172.21.15.139:56672) 2021-11-20T05:59:20.868 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.874+0000 7fbcfdefd700 1 -- 172.21.15.139:0/274282936 learned_addr learned my addr 172.21.15.139:0/274282936 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:20.869 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.875+0000 7fbcfdefd700 1 -- 172.21.15.139:0/274282936 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf400b200 con 0x7fbcf806fbf0 2021-11-20T05:59:20.869 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.875+0000 7fbcfdefd700 1 --2- 172.21.15.139:0/274282936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcf806fbf0 0x7fbcf810ca00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fbcf4006a40 tx=0x7fbcf4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:20.869 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.875+0000 7fbce67fc700 1 -- 172.21.15.139:0/274282936 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcf40114d0 con 0x7fbcf806fbf0 2021-11-20T05:59:20.869 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.875+0000 7fbcfeeff700 1 -- 172.21.15.139:0/274282936 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf810d0f0 con 0x7fbcf806fbf0 2021-11-20T05:59:20.870 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.875+0000 7fbce67fc700 1 -- 172.21.15.139:0/274282936 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcf401b8c0 con 0x7fbcf806fbf0 2021-11-20T05:59:20.870 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.875+0000 7fbce67fc700 1 -- 172.21.15.139:0/274282936 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcf401a8f0 con 0x7fbcf806fbf0 2021-11-20T05:59:20.870 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.875+0000 7fbcfeeff700 1 -- 172.21.15.139:0/274282936 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf810d500 con 0x7fbcf806fbf0 2021-11-20T05:59:20.871 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.876+0000 7fbce67fc700 1 -- 172.21.15.139:0/274282936 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fbcf400f070 con 0x7fbcf806fbf0 2021-11-20T05:59:20.871 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.876+0000 7fbce67fc700 1 --2- 172.21.15.139:0/274282936 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7fbce002fe60 0x7fbce0032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:20.871 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.876+0000 7fbce67fc700 1 -- 172.21.15.139:0/274282936 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbcf4040aa0 con 0x7fbcf806fbf0 2021-11-20T05:59:20.871 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.876+0000 7fbcfd6fc700 1 --2- 172.21.15.139:0/274282936 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7fbce002fe60 0x7fbce0032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:20.872 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.876+0000 7fbcfeeff700 1 -- 172.21.15.139:0/274282936 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf8059100 con 0x7fbcf806fbf0 2021-11-20T05:59:20.872 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.876+0000 7fbcfd6fc700 1 --2- 172.21.15.139:0/274282936 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7fbce002fe60 0x7fbce0032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fbce80096a0 tx=0x7fbce8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:20.872 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:20.880+0000 7fbce67fc700 1 -- 172.21.15.139:0/274282936 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcf4016070 con 0x7fbcf806fbf0 2021-11-20T05:59:21.053 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.065+0000 7fbcfeeff700 1 -- 172.21.15.139:0/274282936 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fbcf8101ba0 con 0x7fbcf806fbf0 2021-11-20T05:59:21.092 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.105+0000 7fbce67fc700 1 -- 172.21.15.139:0/274282936 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fbcf4024460 con 0x7fbcf806fbf0 2021-11-20T05:59:21.093 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.106+0000 7fbce67fc700 1 -- 172.21.15.139:0/274282936 <== mon.0 v2:172.21.15.139:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v12) v1 ==== 88+0+0 (secure 0 0 0) 0x7fbcf4010720 con 0x7fbcf806fbf0 2021-11-20T05:59:21.096 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.108+0000 7fbcfeeff700 1 -- 172.21.15.139:0/274282936 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7fbce002fe60 msgr2=0x7fbce0032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:21.096 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.108+0000 7fbcfeeff700 1 --2- 172.21.15.139:0/274282936 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7fbce002fe60 0x7fbce0032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fbce80096a0 tx=0x7fbce8006b40).stop 2021-11-20T05:59:21.096 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.108+0000 7fbcfeeff700 1 -- 172.21.15.139:0/274282936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcf806fbf0 msgr2=0x7fbcf810ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:21.097 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.108+0000 7fbcfeeff700 1 --2- 172.21.15.139:0/274282936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcf806fbf0 0x7fbcf810ca00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fbcf4006a40 tx=0x7fbcf4006ee0).stop 2021-11-20T05:59:21.097 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.108+0000 7fbcfeeff700 1 -- 172.21.15.139:0/274282936 shutdown_connections 2021-11-20T05:59:21.097 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.108+0000 7fbcfeeff700 1 --2- 172.21.15.139:0/274282936 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7fbce002fe60 0x7fbce0032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:21.098 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.108+0000 7fbcfeeff700 1 --2- 172.21.15.139:0/274282936 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbcf806fbf0 0x7fbcf810ca00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:21.098 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.108+0000 7fbcfeeff700 1 -- 172.21.15.139:0/274282936 >> 172.21.15.139:0/274282936 conn(0x7fbcf80f6110 msgr2=0x7fbcf80f7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:21.098 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.109+0000 7fbcfeeff700 1 -- 172.21.15.139:0/274282936 shutdown_connections 2021-11-20T05:59:21.098 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.109+0000 7fbcfeeff700 1 -- 172.21.15.139:0/274282936 wait complete. 2021-11-20T05:59:21.125 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:21 smithi139 conmon[33501]: audit 2021-11-20T05:59:20.106665+0000 mon.smithi139 ( 2021-11-20T05:59:21.126 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:21 smithi139 conmon[33501]: mon.0) 86 : audit [INF] from='client.? 172.21.15.139:0/824905504' entity='client.admin' 2021-11-20T05:59:21.126 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:21 smithi139 conmon[33501]: audit 2021-11-20T05:59:21.067597+0000 mon.smithi139 (mon. 2021-11-20T05:59:21.127 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:21 smithi139 conmon[33501]: 0) 87 : audit [INF] from='client.? 172.21.15.139:0/274282936' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2021-11-20T05:59:21.446 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2021-11-20T05:59:21.923 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.932+0000 7f956e8bc700 1 Processor -- start 2021-11-20T05:59:21.924 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.932+0000 7f956e8bc700 1 -- start start 2021-11-20T05:59:21.924 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.932+0000 7f956e8bc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9568071cb0 0x7f9568070250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:21.924 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.932+0000 7f956e8bc700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95680720d0 con 0x7f9568071cb0 2021-11-20T05:59:21.925 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.933+0000 7f9567fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9568071cb0 0x7f9568070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:21.925 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.933+0000 7f9567fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9568071cb0 0x7f9568070250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56686/0 (socket says 172.21.15.139:56686) 2021-11-20T05:59:21.925 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.933+0000 7f9567fff700 1 -- 172.21.15.139:0/492735236 learned_addr learned my addr 172.21.15.139:0/492735236 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:21.926 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.933+0000 7f9567fff700 1 -- 172.21.15.139:0/492735236 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9568070740 con 0x7f9568071cb0 2021-11-20T05:59:21.926 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.933+0000 7f9567fff700 1 --2- 172.21.15.139:0/492735236 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9568071cb0 0x7f9568070250 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f955c00b5f0 tx=0x7f955c006ee0).ready entity=mon.0 client_cookie=a6966c0e20c91e1d server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:21.926 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.933+0000 7f9566ffd700 1 -- 172.21.15.139:0/492735236 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f955c00d600 con 0x7f9568071cb0 2021-11-20T05:59:21.926 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.933+0000 7f9566ffd700 1 -- 172.21.15.139:0/492735236 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f955c00dc10 con 0x7f9568071cb0 2021-11-20T05:59:21.927 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.933+0000 7f9566ffd700 1 -- 172.21.15.139:0/492735236 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f955c011cd0 con 0x7f9568071cb0 2021-11-20T05:59:21.927 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.934+0000 7f956e8bc700 1 -- 172.21.15.139:0/492735236 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9568071cb0 msgr2=0x7f9568070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:21.927 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.934+0000 7f956e8bc700 1 --2- 172.21.15.139:0/492735236 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9568071cb0 0x7f9568070250 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f955c00b5f0 tx=0x7f955c006ee0).stop 2021-11-20T05:59:21.927 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.934+0000 7f956e8bc700 1 -- 172.21.15.139:0/492735236 shutdown_connections 2021-11-20T05:59:21.928 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.934+0000 7f956e8bc700 1 --2- 172.21.15.139:0/492735236 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9568071cb0 0x7f9568070250 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:21.928 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.934+0000 7f956e8bc700 1 -- 172.21.15.139:0/492735236 >> 172.21.15.139:0/492735236 conn(0x7f95680f73f0 msgr2=0x7f95680f9810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:21.928 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.934+0000 7f956e8bc700 1 -- 172.21.15.139:0/492735236 shutdown_connections 2021-11-20T05:59:21.928 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.934+0000 7f956e8bc700 1 -- 172.21.15.139:0/492735236 wait complete. 2021-11-20T05:59:21.929 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.935+0000 7f956e8bc700 1 Processor -- start 2021-11-20T05:59:21.929 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.935+0000 7f956e8bc700 1 -- start start 2021-11-20T05:59:21.929 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.935+0000 7f956e8bc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9568071cb0 0x7f956810cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:21.929 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.935+0000 7f956e8bc700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956810d210 con 0x7f9568071cb0 2021-11-20T05:59:21.929 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.935+0000 7f9567fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9568071cb0 0x7f956810cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:21.930 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.935+0000 7f9567fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9568071cb0 0x7f956810cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56688/0 (socket says 172.21.15.139:56688) 2021-11-20T05:59:21.930 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.935+0000 7f9567fff700 1 -- 172.21.15.139:0/378281888 learned_addr learned my addr 172.21.15.139:0/378281888 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:21.931 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.936+0000 7f9567fff700 1 -- 172.21.15.139:0/378281888 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955c00b200 con 0x7f9568071cb0 2021-11-20T05:59:21.932 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.936+0000 7f9567fff700 1 --2- 172.21.15.139:0/378281888 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9568071cb0 0x7f956810cd20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f955c00df90 tx=0x7f955c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:21.932 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.936+0000 7f9564ff9700 1 -- 172.21.15.139:0/378281888 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f955c011da0 con 0x7f9568071cb0 2021-11-20T05:59:21.932 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.936+0000 7f9564ff9700 1 -- 172.21.15.139:0/378281888 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f955c01b8c0 con 0x7f9568071cb0 2021-11-20T05:59:21.932 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.936+0000 7f956e8bc700 1 -- 172.21.15.139:0/378281888 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956810d410 con 0x7f9568071cb0 2021-11-20T05:59:21.933 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.936+0000 7f956e8bc700 1 -- 172.21.15.139:0/378281888 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956810d820 con 0x7f9568071cb0 2021-11-20T05:59:21.933 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.936+0000 7f9564ff9700 1 -- 172.21.15.139:0/378281888 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f955c01a930 con 0x7f9568071cb0 2021-11-20T05:59:21.933 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.937+0000 7f9564ff9700 1 -- 172.21.15.139:0/378281888 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f955c00f070 con 0x7f9568071cb0 2021-11-20T05:59:21.933 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.937+0000 7f9564ff9700 1 --2- 172.21.15.139:0/378281888 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f955402ff00 0x7f95540323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:21.933 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.937+0000 7f9564ff9700 1 -- 172.21.15.139:0/378281888 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f955c042000 con 0x7f9568071cb0 2021-11-20T05:59:21.934 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.937+0000 7f95677fe700 1 -- 172.21.15.139:0/378281888 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f955402ff00 msgr2=0x7f95540323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/3637001875 2021-11-20T05:59:21.934 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.937+0000 7f95677fe700 1 --2- 172.21.15.139:0/378281888 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f955402ff00 0x7f95540323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T05:59:21.934 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.937+0000 7f956e8bc700 1 -- 172.21.15.139:0/378281888 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9548004fa0 con 0x7f9568071cb0 2021-11-20T05:59:21.934 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:21.941+0000 7f9564ff9700 1 -- 172.21.15.139:0/378281888 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f955c024410 con 0x7f9568071cb0 2021-11-20T05:59:22.125 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.138+0000 7f95677fe700 1 -- 172.21.15.139:0/378281888 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f955402ff00 msgr2=0x7f95540323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/3637001875 2021-11-20T05:59:22.125 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.138+0000 7f95677fe700 1 --2- 172.21.15.139:0/378281888 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f955402ff00 0x7f95540323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T05:59:22.137 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.150+0000 7f956e8bc700 1 -- 172.21.15.139:0/378281888 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f9548005e80 con 0x7f9568071cb0 2021-11-20T05:59:22.137 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.150+0000 7f9564ff9700 1 -- 172.21.15.139:0/378281888 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v12) v1 ==== 56+0+104 (secure 0 0 0) 0x7f955c0107d0 con 0x7f9568071cb0 2021-11-20T05:59:22.138 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: { 2021-11-20T05:59:22.138 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "epoch": 12, 2021-11-20T05:59:22.138 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "available": true, 2021-11-20T05:59:22.138 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "active_name": "smithi139.dplyxn", 2021-11-20T05:59:22.139 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "num_standby": 0 2021-11-20T05:59:22.139 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: } 2021-11-20T05:59:22.140 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.152+0000 7f956e8bc700 1 -- 172.21.15.139:0/378281888 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f955402ff00 msgr2=0x7f95540323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T05:59:22.141 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.152+0000 7f956e8bc700 1 --2- 172.21.15.139:0/378281888 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f955402ff00 0x7f95540323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:59:22.141 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.152+0000 7f956e8bc700 1 -- 172.21.15.139:0/378281888 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9568071cb0 msgr2=0x7f956810cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:22.141 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.152+0000 7f956e8bc700 1 --2- 172.21.15.139:0/378281888 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9568071cb0 0x7f956810cd20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f955c00df90 tx=0x7f955c006ee0).stop 2021-11-20T05:59:22.141 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.152+0000 7f956e8bc700 1 -- 172.21.15.139:0/378281888 shutdown_connections 2021-11-20T05:59:22.142 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.152+0000 7f956e8bc700 1 --2- 172.21.15.139:0/378281888 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f955402ff00 0x7f95540323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:59:22.142 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.152+0000 7f956e8bc700 1 --2- 172.21.15.139:0/378281888 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9568071cb0 0x7f956810cd20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:22.142 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.152+0000 7f956e8bc700 1 -- 172.21.15.139:0/378281888 >> 172.21.15.139:0/378281888 conn(0x7f95680f73f0 msgr2=0x7f95680f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:22.142 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.153+0000 7f956e8bc700 1 -- 172.21.15.139:0/378281888 shutdown_connections 2021-11-20T05:59:22.142 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.153+0000 7f956e8bc700 1 -- 172.21.15.139:0/378281888 wait complete. 2021-11-20T05:59:22.489 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for the mgr to restart... 2021-11-20T05:59:22.489 INFO:teuthology.orchestra.run.smithi139.stderr:Waiting for mgr epoch 12... 2021-11-20T05:59:22.490 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2021-11-20T05:59:22.497 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:22 smithi139 conmon[33501]: audit 2021-11-20T05:59:21.107976+0000 mon.smithi139 (mon.0 2021-11-20T05:59:22.498 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:22 smithi139 conmon[33501]: ) 88 : audit [INF] from='client.? 172.21.15.139:0/274282936' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2021-11-20T05:59:22.498 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:22 smithi139 conmon[33501]: cluster 2021-11-20T05:59:21.108029+0000 mon.smithi139 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi139.dplyxn(active, since 7s) 2021-11-20T05:59:22.963 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.973+0000 7f07b3f2a700 1 Processor -- start 2021-11-20T05:59:22.963 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.974+0000 7f07b3f2a700 1 -- start start 2021-11-20T05:59:22.963 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.974+0000 7f07b3f2a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07ac0faae0 0x7f07ac0faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:22.964 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.974+0000 7f07b3f2a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac0fb480 con 0x7f07ac0faae0 2021-11-20T05:59:22.964 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.974+0000 7f07b1cc6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07ac0faae0 0x7f07ac0faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:22.964 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.974+0000 7f07b1cc6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07ac0faae0 0x7f07ac0faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56696/0 (socket says 172.21.15.139:56696) 2021-11-20T05:59:22.965 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.974+0000 7f07b1cc6700 1 -- 172.21.15.139:0/3114820227 learned_addr learned my addr 172.21.15.139:0/3114820227 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:22.965 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.974+0000 7f07b1cc6700 1 -- 172.21.15.139:0/3114820227 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac0fb5c0 con 0x7f07ac0faae0 2021-11-20T05:59:22.967 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.974+0000 7f07b1cc6700 1 --2- 172.21.15.139:0/3114820227 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07ac0faae0 0x7f07ac0faf00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f079c00b5f0 tx=0x7f079c006ee0).ready entity=mon.0 client_cookie=3f8246a385cf1bda server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:22.968 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.974+0000 7f07b0cc4700 1 -- 172.21.15.139:0/3114820227 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f079c00d600 con 0x7f07ac0faae0 2021-11-20T05:59:22.968 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.975+0000 7f07b0cc4700 1 -- 172.21.15.139:0/3114820227 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f079c00dc10 con 0x7f07ac0faae0 2021-11-20T05:59:22.970 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.975+0000 7f07b0cc4700 1 -- 172.21.15.139:0/3114820227 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f079c011cd0 con 0x7f07ac0faae0 2021-11-20T05:59:22.971 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.975+0000 7f07b3f2a700 1 -- 172.21.15.139:0/3114820227 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07ac0faae0 msgr2=0x7f07ac0faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:22.971 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.975+0000 7f07b3f2a700 1 --2- 172.21.15.139:0/3114820227 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07ac0faae0 0x7f07ac0faf00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f079c00b5f0 tx=0x7f079c006ee0).stop 2021-11-20T05:59:22.972 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.975+0000 7f07b3f2a700 1 -- 172.21.15.139:0/3114820227 shutdown_connections 2021-11-20T05:59:22.972 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.975+0000 7f07b3f2a700 1 --2- 172.21.15.139:0/3114820227 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07ac0faae0 0x7f07ac0faf00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:22.973 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.975+0000 7f07b3f2a700 1 -- 172.21.15.139:0/3114820227 >> 172.21.15.139:0/3114820227 conn(0x7f07ac0f6280 msgr2=0x7f07ac0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:22.973 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.976+0000 7f07b3f2a700 1 -- 172.21.15.139:0/3114820227 shutdown_connections 2021-11-20T05:59:22.973 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.976+0000 7f07b3f2a700 1 -- 172.21.15.139:0/3114820227 wait complete. 2021-11-20T05:59:22.976 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.976+0000 7f07b3f2a700 1 Processor -- start 2021-11-20T05:59:22.976 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.976+0000 7f07b3f2a700 1 -- start start 2021-11-20T05:59:22.977 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.976+0000 7f07b3f2a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07ac0faae0 0x7f07ac100c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:22.977 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.976+0000 7f07b3f2a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac0ff0d0 con 0x7f07ac0faae0 2021-11-20T05:59:22.977 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.976+0000 7f07b1cc6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07ac0faae0 0x7f07ac100c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:22.978 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.976+0000 7f07b1cc6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07ac0faae0 0x7f07ac100c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56698/0 (socket says 172.21.15.139:56698) 2021-11-20T05:59:22.978 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.976+0000 7f07b1cc6700 1 -- 172.21.15.139:0/1863718824 learned_addr learned my addr 172.21.15.139:0/1863718824 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:22.978 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.977+0000 7f07b1cc6700 1 -- 172.21.15.139:0/1863718824 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079c00b200 con 0x7f07ac0faae0 2021-11-20T05:59:22.979 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.977+0000 7f07b1cc6700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07ac0faae0 0x7f07ac100c50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f079c00dfe0 tx=0x7f079c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:22.979 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.977+0000 7f079a7fc700 1 -- 172.21.15.139:0/1863718824 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f079c011df0 con 0x7f07ac0faae0 2021-11-20T05:59:22.980 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.977+0000 7f079a7fc700 1 -- 172.21.15.139:0/1863718824 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f079c01b8c0 con 0x7f07ac0faae0 2021-11-20T05:59:22.980 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.977+0000 7f07b3f2a700 1 -- 172.21.15.139:0/1863718824 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07ac0ff270 con 0x7f07ac0faae0 2021-11-20T05:59:22.981 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.977+0000 7f07b3f2a700 1 -- 172.21.15.139:0/1863718824 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ac0ff680 con 0x7f07ac0faae0 2021-11-20T05:59:22.981 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.977+0000 7f079a7fc700 1 -- 172.21.15.139:0/1863718824 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f079c01a840 con 0x7f07ac0faae0 2021-11-20T05:59:22.981 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.978+0000 7f079a7fc700 1 -- 172.21.15.139:0/1863718824 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f079c00f070 con 0x7f07ac0faae0 2021-11-20T05:59:22.982 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.978+0000 7f079a7fc700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f079402feb0 0x7f0794032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:22.982 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.978+0000 7f07b3f2a700 1 -- 172.21.15.139:0/1863718824 --> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f078c000d10 con 0x7f079402feb0 2021-11-20T05:59:22.982 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.978+0000 7f079a7fc700 1 -- 172.21.15.139:0/1863718824 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f079c041f40 con 0x7f07ac0faae0 2021-11-20T05:59:22.983 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.978+0000 7f07b14c5700 1 -- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f079402feb0 msgr2=0x7f0794032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/3637001875 2021-11-20T05:59:22.983 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:22.978+0000 7f07b14c5700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f079402feb0 0x7f0794032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T05:59:23.165 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:23.179+0000 7f07b14c5700 1 -- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f079402feb0 msgr2=0x7f0794032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/3637001875 2021-11-20T05:59:23.166 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:23.179+0000 7f07b14c5700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f079402feb0 0x7f0794032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T05:59:23.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:23 smithi139 conmon[33501]: audit 2021-11-20T05:59:22.151469+0000 mon.smithi139 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.139:0/378281888' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-11-20T05:59:23.567 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:23.580+0000 7f07b14c5700 1 -- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f079402feb0 msgr2=0x7f0794032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/3637001875 2021-11-20T05:59:23.567 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:23.580+0000 7f07b14c5700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f079402feb0 0x7f0794032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T05:59:24.367 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:24.381+0000 7f07b14c5700 1 -- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f079402feb0 msgr2=0x7f0794032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/3637001875 2021-11-20T05:59:24.368 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:24.381+0000 7f07b14c5700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f079402feb0 0x7f0794032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-20T05:59:25.969 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:25.982+0000 7f07b14c5700 1 -- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f079402feb0 msgr2=0x7f0794032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.139:6800/3637001875 2021-11-20T05:59:25.970 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:25.982+0000 7f07b14c5700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f079402feb0 0x7f0794032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-20T05:59:26.154 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:26.166+0000 7f079a7fc700 1 -- 172.21.15.139:0/1863718824 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mgrmap(e 13) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f079c010dd0 con 0x7f07ac0faae0 2021-11-20T05:59:26.154 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:26.166+0000 7f079a7fc700 1 -- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f079402feb0 msgr2=0x7f0794032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T05:59:26.155 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:26.166+0000 7f079a7fc700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f079402feb0 0x7f0794032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:59:26.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:26 smithi139 conmon[33501]: cluster 2021-11-20T05:59:26.112517+0000 mon.smithi139 (mon. 2021-11-20T05:59:26.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:26 smithi139 conmon[33501]: 0) 91 : cluster [INF] Active manager daemon smithi139.dplyxn restarted 2021-11-20T05:59:26.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:26 smithi139 conmon[33501]: cluster 2021-11-20T05:59:26.112918+0000 mon.smithi139 (mon.0) 92 : cluster [INF] Activating manager daemon smithi139.dplyxn 2021-11-20T05:59:26.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:26 smithi139 conmon[33501]: cluster 2021-11-20T05:59:26.117175+0000 mon.smithi139 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-11-20T05:59:27.163 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.174+0000 7f079a7fc700 1 -- 172.21.15.139:0/1863718824 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f079c0423a0 con 0x7f07ac0faae0 2021-11-20T05:59:27.163 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.174+0000 7f079a7fc700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0794033990 0x7f0794035d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:27.164 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.174+0000 7f079a7fc700 1 -- 172.21.15.139:0/1863718824 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f078c000d10 con 0x7f0794033990 2021-11-20T05:59:27.164 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.175+0000 7f07b14c5700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0794033990 0x7f0794035d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:27.164 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.175+0000 7f07b14c5700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0794033990 0x7f0794035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f07a8006fd0 tx=0x7f07a8006cf0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:27.164 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.176+0000 7f079a7fc700 1 -- 172.21.15.139:0/1863718824 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f078c000d10 con 0x7f0794033990 2021-11-20T05:59:27.168 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.182+0000 7f07b3f2a700 1 -- 172.21.15.139:0/1863718824 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f078c002430 con 0x7f0794033990 2021-11-20T05:59:27.170 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.182+0000 7f079a7fc700 1 -- 172.21.15.139:0/1863718824 <== mgr.14162 v2:172.21.15.139:6800/2363962993 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f078c002430 con 0x7f0794033990 2021-11-20T05:59:27.170 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: { 2021-11-20T05:59:27.170 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "mgrmap_epoch": 14, 2021-11-20T05:59:27.170 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: "initialized": true 2021-11-20T05:59:27.171 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: } 2021-11-20T05:59:27.171 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.182+0000 7f07b3f2a700 1 -- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0794033990 msgr2=0x7f0794035d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:27.171 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.182+0000 7f07b3f2a700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0794033990 0x7f0794035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f07a8006fd0 tx=0x7f07a8006cf0).stop 2021-11-20T05:59:27.172 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.182+0000 7f07b3f2a700 1 -- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07ac0faae0 msgr2=0x7f07ac100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:27.172 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.182+0000 7f07b3f2a700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07ac0faae0 0x7f07ac100c50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f079c00dfe0 tx=0x7f079c006ee0).stop 2021-11-20T05:59:27.172 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.182+0000 7f07b3f2a700 1 -- 172.21.15.139:0/1863718824 shutdown_connections 2021-11-20T05:59:27.172 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.182+0000 7f07b3f2a700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0794033990 0x7f0794035d80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:27.173 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.182+0000 7f07b3f2a700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07ac0faae0 0x7f07ac100c50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:27.173 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.182+0000 7f07b3f2a700 1 --2- 172.21.15.139:0/1863718824 >> [v2:172.21.15.139:6800/3637001875,v1:172.21.15.139:6801/3637001875] conn(0x7f079402feb0 0x7f0794032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T05:59:27.173 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.182+0000 7f07b3f2a700 1 -- 172.21.15.139:0/1863718824 >> 172.21.15.139:0/1863718824 conn(0x7f07ac0f6280 msgr2=0x7f07ac0f6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:27.173 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.182+0000 7f07b3f2a700 1 -- 172.21.15.139:0/1863718824 shutdown_connections 2021-11-20T05:59:27.173 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:27.182+0000 7f07b3f2a700 1 -- 172.21.15.139:0/1863718824 wait complete. 2021-11-20T05:59:27.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:27 smithi139 conmon[33501]: cluster 2021-11-20T05:59:26.168773+0000 mon.smithi139 (mon.0) 94 2021-11-20T05:59:27.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:27 smithi139 conmon[33501]: : cluster [DBG] mgrmap e13: smithi139.dplyxn(active, starting, since 0.055944s) 2021-11-20T05:59:27.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:27 smithi139 conmon[33501]: audit 2021-11-20T05:59:26.169993+0000 mon.smithi139 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi139"}]: dispatch 2021-11-20T05:59:27.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:27 smithi139 conmon[33501]: audit 2021-11-20T05:59:26.170558+0000 mon.smithi139 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mgr metadata", "who": "smithi139.dplyxn", "id": "smithi139.dplyxn"}]: dispatch 2021-11-20T05:59:27.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:27 smithi139 conmon[33501]: audit 2021-11-20T05:59:26.171363+0000 mon.smithi139 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-20T05:59:27.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:27 smithi139 conmon[33501]: audit 2021-11-20T05:59:26.171482+0000 mon.smithi139 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-20T05:59:27.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:27 smithi139 conmon[33501]: audit 2021-11-20T05:59:26.171599+0000 mon.smithi139 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-20T05:59:27.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:27 smithi139 conmon[33501]: cluster 2021-11-20T05:59:26.185026+0000 mon.smithi139 (mon.0) 100 : cluster [INF] Manager daemon smithi139.dplyxn is now available 2021-11-20T05:59:27.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:27 smithi139 conmon[33501]: audit 2021-11-20T05:59:26.281638+0000 mon.smithi139 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:59:27.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:27 smithi139 conmon[33501]: audit 2021-11-20T05:59:26.283760+0000 mon.smithi139 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T05:59:27.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:27 smithi139 conmon[33501]: audit 2021-11-20T05:59:26.300488+0000 mon.smithi139 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T05:59:27.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:27 smithi139 conmon[33501]: audit 2021-11-20T05:59:26.309352+0000 mon.smithi139 (mon.0 2021-11-20T05:59:27.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:27 smithi139 conmon[33501]: ) 104 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T05:59:27.638 INFO:teuthology.orchestra.run.smithi139.stderr:mgr epoch 12 is available 2021-11-20T05:59:27.638 INFO:teuthology.orchestra.run.smithi139.stderr:Generating a dashboard self-signed certificate... 2021-11-20T05:59:27.638 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2021-11-20T05:59:28.171 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.184+0000 7f98bee06700 1 Processor -- start 2021-11-20T05:59:28.172 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.184+0000 7f98bee06700 1 -- start start 2021-11-20T05:59:28.173 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.184+0000 7f98bee06700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98b806fc30 0x7f98b8100c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:28.174 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.184+0000 7f98bee06700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b80fc6c0 con 0x7f98b806fc30 2021-11-20T05:59:28.174 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.185+0000 7f98bcba2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98b806fc30 0x7f98b8100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:28.174 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.185+0000 7f98bcba2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98b806fc30 0x7f98b8100c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56740/0 (socket says 172.21.15.139:56740) 2021-11-20T05:59:28.174 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.185+0000 7f98bcba2700 1 -- 172.21.15.139:0/3163576941 learned_addr learned my addr 172.21.15.139:0/3163576941 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:28.175 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.185+0000 7f98bcba2700 1 -- 172.21.15.139:0/3163576941 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b8101130 con 0x7f98b806fc30 2021-11-20T05:59:28.175 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.185+0000 7f98bcba2700 1 --2- 172.21.15.139:0/3163576941 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98b806fc30 0x7f98b8100c40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f98ac00b5f0 tx=0x7f98ac006ee0).ready entity=mon.0 client_cookie=adfe32ca5f7b0db3 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:28.175 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.185+0000 7f98b77fe700 1 -- 172.21.15.139:0/3163576941 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98ac00d600 con 0x7f98b806fc30 2021-11-20T05:59:28.175 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.186+0000 7f98b77fe700 1 -- 172.21.15.139:0/3163576941 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98ac00dc10 con 0x7f98b806fc30 2021-11-20T05:59:28.176 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.186+0000 7f98b77fe700 1 -- 172.21.15.139:0/3163576941 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98ac011cd0 con 0x7f98b806fc30 2021-11-20T05:59:28.176 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.186+0000 7f98bee06700 1 -- 172.21.15.139:0/3163576941 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98b806fc30 msgr2=0x7f98b8100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:28.176 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.186+0000 7f98bee06700 1 --2- 172.21.15.139:0/3163576941 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98b806fc30 0x7f98b8100c40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f98ac00b5f0 tx=0x7f98ac006ee0).stop 2021-11-20T05:59:28.176 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.186+0000 7f98bee06700 1 -- 172.21.15.139:0/3163576941 shutdown_connections 2021-11-20T05:59:28.176 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.186+0000 7f98bee06700 1 --2- 172.21.15.139:0/3163576941 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98b806fc30 0x7f98b8100c40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:28.177 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.186+0000 7f98bee06700 1 -- 172.21.15.139:0/3163576941 >> 172.21.15.139:0/3163576941 conn(0x7f98b80f6280 msgr2=0x7f98b80f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:28.177 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.186+0000 7f98bee06700 1 -- 172.21.15.139:0/3163576941 shutdown_connections 2021-11-20T05:59:28.177 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.186+0000 7f98bee06700 1 -- 172.21.15.139:0/3163576941 wait complete. 2021-11-20T05:59:28.177 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.187+0000 7f98bee06700 1 Processor -- start 2021-11-20T05:59:28.178 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.187+0000 7f98bee06700 1 -- start start 2021-11-20T05:59:28.178 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.187+0000 7f98bee06700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98b806fc30 0x7f98b810cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:28.178 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.187+0000 7f98bee06700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b810d200 con 0x7f98b806fc30 2021-11-20T05:59:28.178 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.187+0000 7f98bcba2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98b806fc30 0x7f98b810cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:28.178 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.187+0000 7f98bcba2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98b806fc30 0x7f98b810cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56742/0 (socket says 172.21.15.139:56742) 2021-11-20T05:59:28.179 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.187+0000 7f98bcba2700 1 -- 172.21.15.139:0/1081999688 learned_addr learned my addr 172.21.15.139:0/1081999688 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:28.180 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.188+0000 7f98bcba2700 1 -- 172.21.15.139:0/1081999688 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac00b200 con 0x7f98b806fc30 2021-11-20T05:59:28.181 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.188+0000 7f98bcba2700 1 --2- 172.21.15.139:0/1081999688 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98b806fc30 0x7f98b810cd10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f98ac00dfb0 tx=0x7f98ac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:28.181 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.188+0000 7f98b57fa700 1 -- 172.21.15.139:0/1081999688 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98ac011df0 con 0x7f98b806fc30 2021-11-20T05:59:28.181 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.188+0000 7f98b57fa700 1 -- 172.21.15.139:0/1081999688 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98ac01b8c0 con 0x7f98b806fc30 2021-11-20T05:59:28.181 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.188+0000 7f98bee06700 1 -- 172.21.15.139:0/1081999688 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b810d400 con 0x7f98b806fc30 2021-11-20T05:59:28.182 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.188+0000 7f98b57fa700 1 -- 172.21.15.139:0/1081999688 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98ac01a800 con 0x7f98b806fc30 2021-11-20T05:59:28.182 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.188+0000 7f98bee06700 1 -- 172.21.15.139:0/1081999688 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b810d810 con 0x7f98b806fc30 2021-11-20T05:59:28.182 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.189+0000 7f98b57fa700 1 -- 172.21.15.139:0/1081999688 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f98ac00f070 con 0x7f98b806fc30 2021-11-20T05:59:28.182 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.189+0000 7f98b57fa700 1 --2- 172.21.15.139:0/1081999688 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f989c02fd70 0x7f989c032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:28.183 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.189+0000 7f98b57fa700 1 -- 172.21.15.139:0/1081999688 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98ac01ba30 con 0x7f98b806fc30 2021-11-20T05:59:28.183 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.189+0000 7f98b7fff700 1 --2- 172.21.15.139:0/1081999688 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f989c02fd70 0x7f989c032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:28.183 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.189+0000 7f98bee06700 1 -- 172.21.15.139:0/1081999688 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b8101e70 con 0x7f98b806fc30 2021-11-20T05:59:28.183 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.190+0000 7f98b7fff700 1 --2- 172.21.15.139:0/1081999688 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f989c02fd70 0x7f989c032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f98a40096a0 tx=0x7f98a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:28.184 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.194+0000 7f98b57fa700 1 -- 172.21.15.139:0/1081999688 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98ac016030 con 0x7f98b806fc30 2021-11-20T05:59:28.344 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.357+0000 7f98bee06700 1 -- 172.21.15.139:0/1081999688 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f98b8102030 con 0x7f989c02fd70 2021-11-20T05:59:28.410 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.424+0000 7f98b57fa700 1 -- 172.21.15.139:0/1081999688 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f98b8102030 con 0x7f989c02fd70 2021-11-20T05:59:28.411 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: Self-signed certificate created 2021-11-20T05:59:28.413 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.426+0000 7f98bee06700 1 -- 172.21.15.139:0/1081999688 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f989c02fd70 msgr2=0x7f989c032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:28.413 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.426+0000 7f98bee06700 1 --2- 172.21.15.139:0/1081999688 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f989c02fd70 0x7f989c032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f98a40096a0 tx=0x7f98a4006b40).stop 2021-11-20T05:59:28.413 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.426+0000 7f98bee06700 1 -- 172.21.15.139:0/1081999688 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98b806fc30 msgr2=0x7f98b810cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:28.413 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.426+0000 7f98bee06700 1 --2- 172.21.15.139:0/1081999688 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98b806fc30 0x7f98b810cd10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f98ac00dfb0 tx=0x7f98ac006ee0).stop 2021-11-20T05:59:28.414 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.426+0000 7f98bee06700 1 -- 172.21.15.139:0/1081999688 shutdown_connections 2021-11-20T05:59:28.414 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.426+0000 7f98bee06700 1 --2- 172.21.15.139:0/1081999688 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f989c02fd70 0x7f989c032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:28.414 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.426+0000 7f98bee06700 1 --2- 172.21.15.139:0/1081999688 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98b806fc30 0x7f98b810cd10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:28.414 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.426+0000 7f98bee06700 1 -- 172.21.15.139:0/1081999688 >> 172.21.15.139:0/1081999688 conn(0x7f98b80f6280 msgr2=0x7f98b80f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:28.415 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.426+0000 7f98bee06700 1 -- 172.21.15.139:0/1081999688 shutdown_connections 2021-11-20T05:59:28.415 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:28.426+0000 7f98bee06700 1 -- 172.21.15.139:0/1081999688 wait complete. 2021-11-20T05:59:28.415 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:28 smithi139 conmon[33501]: audit 2021-11-20T05:59:27.176456+0000 mgr.smithi139.dplyxn (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-11-20T05:59:28.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:28 smithi139 conmon[33501]: cluster 2021-11-20 2021-11-20T05:59:28.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:28 smithi139 conmon[33501]: T05:59:27.177296+0000 mon.smithi139 (mon.0) 105 : cluster [DBG] mgrmap e14: smithi139.dplyxn(active, since 1.06446s) 2021-11-20T05:59:28.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:28 smithi139 conmon[33501]: audit 2021-11-20T05:59:27.182945+0000 mgr.smithi139.dplyxn (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-11-20T05:59:28.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:28 smithi139 conmon[33501]: audit 2021-11-20T05:59:28.063239+0000 mon.smithi139 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:28.828 INFO:teuthology.orchestra.run.smithi139.stderr:Creating initial admin user... 2021-11-20T05:59:28.829 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_1jm4n3y:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2021-11-20T05:59:29.454 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.465+0000 7f3dee5f0700 1 Processor -- start 2021-11-20T05:59:29.455 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.465+0000 7f3dee5f0700 1 -- start start 2021-11-20T05:59:29.455 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.465+0000 7f3dee5f0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de806fbf0 0x7f3de8100a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:29.455 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.465+0000 7f3dee5f0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de80fc430 con 0x7f3de806fbf0 2021-11-20T05:59:29.455 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.465+0000 7f3ded5ee700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de806fbf0 0x7f3de8100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:29.456 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.465+0000 7f3ded5ee700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de806fbf0 0x7f3de8100a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56746/0 (socket says 172.21.15.139:56746) 2021-11-20T05:59:29.456 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.465+0000 7f3ded5ee700 1 -- 172.21.15.139:0/3873010340 learned_addr learned my addr 172.21.15.139:0/3873010340 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:29.456 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.465+0000 7f3ded5ee700 1 -- 172.21.15.139:0/3873010340 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de8100f30 con 0x7f3de806fbf0 2021-11-20T05:59:29.456 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.466+0000 7f3ded5ee700 1 --2- 172.21.15.139:0/3873010340 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de806fbf0 0x7f3de8100a40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3de400b5f0 tx=0x7f3de4006ee0).ready entity=mon.0 client_cookie=c113c83cef3168a server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:29.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.466+0000 7f3dd7fff700 1 -- 172.21.15.139:0/3873010340 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3de400d600 con 0x7f3de806fbf0 2021-11-20T05:59:29.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.466+0000 7f3dd7fff700 1 -- 172.21.15.139:0/3873010340 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3de400dc10 con 0x7f3de806fbf0 2021-11-20T05:59:29.457 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.466+0000 7f3dd7fff700 1 -- 172.21.15.139:0/3873010340 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3de4011d00 con 0x7f3de806fbf0 2021-11-20T05:59:29.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.466+0000 7f3dee5f0700 1 -- 172.21.15.139:0/3873010340 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de806fbf0 msgr2=0x7f3de8100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:29.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.466+0000 7f3dee5f0700 1 --2- 172.21.15.139:0/3873010340 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de806fbf0 0x7f3de8100a40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3de400b5f0 tx=0x7f3de4006ee0).stop 2021-11-20T05:59:29.458 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.466+0000 7f3dee5f0700 1 -- 172.21.15.139:0/3873010340 shutdown_connections 2021-11-20T05:59:29.459 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.466+0000 7f3dee5f0700 1 --2- 172.21.15.139:0/3873010340 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de806fbf0 0x7f3de8100a40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:29.459 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.466+0000 7f3dee5f0700 1 -- 172.21.15.139:0/3873010340 >> 172.21.15.139:0/3873010340 conn(0x7f3de80f60e0 msgr2=0x7f3de80f8500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:29.460 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.467+0000 7f3dee5f0700 1 -- 172.21.15.139:0/3873010340 shutdown_connections 2021-11-20T05:59:29.460 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.467+0000 7f3dee5f0700 1 -- 172.21.15.139:0/3873010340 wait complete. 2021-11-20T05:59:29.461 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.467+0000 7f3dee5f0700 1 Processor -- start 2021-11-20T05:59:29.461 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.468+0000 7f3dee5f0700 1 -- start start 2021-11-20T05:59:29.462 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.468+0000 7f3dee5f0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de806fbf0 0x7f3de810cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:29.462 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.468+0000 7f3dee5f0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de810d000 con 0x7f3de806fbf0 2021-11-20T05:59:29.462 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.468+0000 7f3ded5ee700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de806fbf0 0x7f3de810cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:29.463 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.468+0000 7f3ded5ee700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de806fbf0 0x7f3de810cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56748/0 (socket says 172.21.15.139:56748) 2021-11-20T05:59:29.465 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.468+0000 7f3ded5ee700 1 -- 172.21.15.139:0/1044354278 learned_addr learned my addr 172.21.15.139:0/1044354278 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:29.465 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.468+0000 7f3ded5ee700 1 -- 172.21.15.139:0/1044354278 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de400b200 con 0x7f3de806fbf0 2021-11-20T05:59:29.466 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.468+0000 7f3ded5ee700 1 --2- 172.21.15.139:0/1044354278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de806fbf0 0x7f3de810cb10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3de400b160 tx=0x7f3de4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:29.466 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.468+0000 7f3dd5ffb700 1 -- 172.21.15.139:0/1044354278 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3de4011e20 con 0x7f3de806fbf0 2021-11-20T05:59:29.466 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.468+0000 7f3dd5ffb700 1 -- 172.21.15.139:0/1044354278 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3de401b8c0 con 0x7f3de806fbf0 2021-11-20T05:59:29.467 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.468+0000 7f3dee5f0700 1 -- 172.21.15.139:0/1044354278 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de810d200 con 0x7f3de806fbf0 2021-11-20T05:59:29.467 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.468+0000 7f3dd5ffb700 1 -- 172.21.15.139:0/1044354278 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3de401a790 con 0x7f3de806fbf0 2021-11-20T05:59:29.467 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.469+0000 7f3dee5f0700 1 -- 172.21.15.139:0/1044354278 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de810d610 con 0x7f3de806fbf0 2021-11-20T05:59:29.468 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.469+0000 7f3dd5ffb700 1 -- 172.21.15.139:0/1044354278 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f3de400f070 con 0x7f3de806fbf0 2021-11-20T05:59:29.468 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.469+0000 7f3dd5ffb700 1 --2- 172.21.15.139:0/1044354278 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3dd002ff00 0x7f3dd00323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:29.468 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.469+0000 7f3dd5ffb700 1 -- 172.21.15.139:0/1044354278 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3de4041180 con 0x7f3de806fbf0 2021-11-20T05:59:29.469 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.470+0000 7f3decded700 1 --2- 172.21.15.139:0/1044354278 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3dd002ff00 0x7f3dd00323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:29.469 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.470+0000 7f3decded700 1 --2- 172.21.15.139:0/1044354278 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3dd002ff00 0x7f3dd00323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3dd80096a0 tx=0x7f3dd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:29.469 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.470+0000 7f3dee5f0700 1 -- 172.21.15.139:0/1044354278 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de8059100 con 0x7f3de806fbf0 2021-11-20T05:59:29.469 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.474+0000 7f3dd5ffb700 1 -- 172.21.15.139:0/1044354278 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3de4016070 con 0x7f3de806fbf0 2021-11-20T05:59:29.628 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:29 smithi139 conmon[33501]: audit 2021-11-20T05:59:28.358697+0000 mgr.smithi139.dplyxn (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:59:29.628 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:29 smithi139 conmon[33501]: audit 2021-11-20T05:59:28 2021-11-20T05:59:29.628 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:29 smithi139 conmon[33501]: .422687+0000 mon.smithi139 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:29.629 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:29 smithi139 conmon[33501]: audit 2021-11-20T05:59:28.424524+0000 mon.smithi139 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:29.629 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:29 smithi139 conmon[33501]: cluster 2021-11-20T05:59:29.064741+0000 mon.smithi139 (mon.0) 109 : cluster [DBG] mgrmap e15: smithi139.dplyxn(active, since 2s) 2021-11-20T05:59:29.629 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.641+0000 7f3dee5f0700 1 -- 172.21.15.139:0/1044354278 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3de8045ad0 con 0x7f3dd002ff00 2021-11-20T05:59:29.846 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.859+0000 7f3dd5ffb700 1 -- 172.21.15.139:0/1044354278 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f3de8045ad0 con 0x7f3dd002ff00 2021-11-20T05:59:29.846 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$ajXCEi9YtPJ.eLyvIRGs2uQ8Ue3N5xvwst2z8ZbzINw1tSpPjQ9Ci", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1637387969, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2021-11-20T05:59:29.849 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.861+0000 7f3dee5f0700 1 -- 172.21.15.139:0/1044354278 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3dd002ff00 msgr2=0x7f3dd00323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:29.850 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.861+0000 7f3dee5f0700 1 --2- 172.21.15.139:0/1044354278 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3dd002ff00 0x7f3dd00323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3dd80096a0 tx=0x7f3dd8006b40).stop 2021-11-20T05:59:29.850 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.861+0000 7f3dee5f0700 1 -- 172.21.15.139:0/1044354278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de806fbf0 msgr2=0x7f3de810cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:29.853 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.861+0000 7f3dee5f0700 1 --2- 172.21.15.139:0/1044354278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de806fbf0 0x7f3de810cb10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3de400b160 tx=0x7f3de4006ee0).stop 2021-11-20T05:59:29.853 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.861+0000 7f3dee5f0700 1 -- 172.21.15.139:0/1044354278 shutdown_connections 2021-11-20T05:59:29.853 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.861+0000 7f3dee5f0700 1 --2- 172.21.15.139:0/1044354278 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3dd002ff00 0x7f3dd00323c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:29.853 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.861+0000 7f3dee5f0700 1 --2- 172.21.15.139:0/1044354278 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3de806fbf0 0x7f3de810cb10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:29.854 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.861+0000 7f3dee5f0700 1 -- 172.21.15.139:0/1044354278 >> 172.21.15.139:0/1044354278 conn(0x7f3de80f60e0 msgr2=0x7f3de80f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:29.854 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.862+0000 7f3dee5f0700 1 -- 172.21.15.139:0/1044354278 shutdown_connections 2021-11-20T05:59:29.854 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:29.862+0000 7f3dee5f0700 1 -- 172.21.15.139:0/1044354278 wait complete. 2021-11-20T05:59:30.167 INFO:teuthology.orchestra.run.smithi139.stderr:Fetching dashboard port number... 2021-11-20T05:59:30.168 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi139 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpz2uy55_5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnvkdejxu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2021-11-20T05:59:31.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:30 smithi139 conmon[33501]: audit 2021-11-20T05:59:29.643206+0000 mgr.smithi139.dplyxn (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:59:31.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:30 smithi139 conmon[33501]: audit 2021-11-20T05:59:29.860342+0000 mon.smithi139 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:31.604 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.613+0000 7fdcb3f0d700 1 Processor -- start 2021-11-20T05:59:31.604 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.613+0000 7fdcb3f0d700 1 -- start start 2021-11-20T05:59:31.605 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.613+0000 7fdcb3f0d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcac0fd2c0 0x7fdcac0fd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:31.605 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.613+0000 7fdcb3f0d700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac0fdbd0 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.605 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.614+0000 7fdcb1ca9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcac0fd2c0 0x7fdcac0fd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:31.605 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.614+0000 7fdcb1ca9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcac0fd2c0 0x7fdcac0fd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56756/0 (socket says 172.21.15.139:56756) 2021-11-20T05:59:31.606 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.614+0000 7fdcb1ca9700 1 -- 172.21.15.139:0/200020169 learned_addr learned my addr 172.21.15.139:0/200020169 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:31.606 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.614+0000 7fdcb1ca9700 1 -- 172.21.15.139:0/200020169 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac0fdd10 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.606 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.614+0000 7fdcb1ca9700 1 --2- 172.21.15.139:0/200020169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcac0fd2c0 0x7fdcac0fd6e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fdca800b5f0 tx=0x7fdca8006ee0).ready entity=mon.0 client_cookie=b0e7a39f2bb26b9c server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:31.606 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.614+0000 7fdcb0ca7700 1 -- 172.21.15.139:0/200020169 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdca800d600 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.607 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.614+0000 7fdcb0ca7700 1 -- 172.21.15.139:0/200020169 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdca800dc10 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.607 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.614+0000 7fdcb0ca7700 1 -- 172.21.15.139:0/200020169 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdca8011cd0 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.607 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.615+0000 7fdcb3f0d700 1 -- 172.21.15.139:0/200020169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcac0fd2c0 msgr2=0x7fdcac0fd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:31.607 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.615+0000 7fdcb3f0d700 1 --2- 172.21.15.139:0/200020169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcac0fd2c0 0x7fdcac0fd6e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fdca800b5f0 tx=0x7fdca8006ee0).stop 2021-11-20T05:59:31.607 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.615+0000 7fdcb3f0d700 1 -- 172.21.15.139:0/200020169 shutdown_connections 2021-11-20T05:59:31.608 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.615+0000 7fdcb3f0d700 1 --2- 172.21.15.139:0/200020169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcac0fd2c0 0x7fdcac0fd6e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:31.608 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.615+0000 7fdcb3f0d700 1 -- 172.21.15.139:0/200020169 >> 172.21.15.139:0/200020169 conn(0x7fdcac0f5a80 msgr2=0x7fdcac0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:31.608 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.615+0000 7fdcb3f0d700 1 -- 172.21.15.139:0/200020169 shutdown_connections 2021-11-20T05:59:31.608 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.615+0000 7fdcb3f0d700 1 -- 172.21.15.139:0/200020169 wait complete. 2021-11-20T05:59:31.609 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.615+0000 7fdcb3f0d700 1 Processor -- start 2021-11-20T05:59:31.609 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.615+0000 7fdcb3f0d700 1 -- start start 2021-11-20T05:59:31.609 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.616+0000 7fdcb3f0d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcac0fd2c0 0x7fdcac0fcb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:31.609 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.616+0000 7fdcb3f0d700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac0fb010 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.610 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.616+0000 7fdcb1ca9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcac0fd2c0 0x7fdcac0fcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:31.610 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.616+0000 7fdcb1ca9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcac0fd2c0 0x7fdcac0fcb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56758/0 (socket says 172.21.15.139:56758) 2021-11-20T05:59:31.610 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.616+0000 7fdcb1ca9700 1 -- 172.21.15.139:0/1522819457 learned_addr learned my addr 172.21.15.139:0/1522819457 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:31.610 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.616+0000 7fdcb1ca9700 1 -- 172.21.15.139:0/1522819457 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca800b200 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.611 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.616+0000 7fdcb1ca9700 1 --2- 172.21.15.139:0/1522819457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcac0fd2c0 0x7fdcac0fcb90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fdca8006cb0 tx=0x7fdca8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:31.611 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.616+0000 7fdc9a7fc700 1 -- 172.21.15.139:0/1522819457 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdca8011df0 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.611 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.616+0000 7fdc9a7fc700 1 -- 172.21.15.139:0/1522819457 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdca801b8c0 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.611 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.616+0000 7fdc9a7fc700 1 -- 172.21.15.139:0/1522819457 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdca801a930 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.611 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.616+0000 7fdcb3f0d700 1 -- 172.21.15.139:0/1522819457 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcac0fb1b0 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.612 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.616+0000 7fdcb3f0d700 1 -- 172.21.15.139:0/1522819457 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcac0fb5c0 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.612 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.617+0000 7fdc9a7fc700 1 -- 172.21.15.139:0/1522819457 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fdca800f070 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.612 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.617+0000 7fdc9a7fc700 1 --2- 172.21.15.139:0/1522819457 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdc9402ff00 0x7fdc940323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:31.612 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.617+0000 7fdc9a7fc700 1 -- 172.21.15.139:0/1522819457 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdca8041000 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.613 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.617+0000 7fdcb14a8700 1 --2- 172.21.15.139:0/1522819457 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdc9402ff00 0x7fdc940323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:31.613 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.618+0000 7fdcb3f0d700 1 -- 172.21.15.139:0/1522819457 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcac059100 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.613 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.618+0000 7fdcb14a8700 1 --2- 172.21.15.139:0/1522819457 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdc9402ff00 0x7fdc940323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fdc9c0096a0 tx=0x7fdc9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:31.614 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.622+0000 7fdc9a7fc700 1 -- 172.21.15.139:0/1522819457 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdca8016070 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.760 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.773+0000 7fdcb3f0d700 1 -- 172.21.15.139:0/1522819457 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fdcac045ad0 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.761 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.773+0000 7fdc9a7fc700 1 -- 172.21.15.139:0/1522819457 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7fdca8019950 con 0x7fdcac0fd2c0 2021-11-20T05:59:31.761 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 8443 2021-11-20T05:59:31.763 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.775+0000 7fdcb3f0d700 1 -- 172.21.15.139:0/1522819457 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdc9402ff00 msgr2=0x7fdc940323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:31.764 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.775+0000 7fdcb3f0d700 1 --2- 172.21.15.139:0/1522819457 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdc9402ff00 0x7fdc940323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fdc9c0096a0 tx=0x7fdc9c006b40).stop 2021-11-20T05:59:31.764 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.775+0000 7fdcb3f0d700 1 -- 172.21.15.139:0/1522819457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcac0fd2c0 msgr2=0x7fdcac0fcb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:31.764 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.775+0000 7fdcb3f0d700 1 --2- 172.21.15.139:0/1522819457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcac0fd2c0 0x7fdcac0fcb90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fdca8006cb0 tx=0x7fdca8006ee0).stop 2021-11-20T05:59:31.764 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.775+0000 7fdcb3f0d700 1 -- 172.21.15.139:0/1522819457 shutdown_connections 2021-11-20T05:59:31.765 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.775+0000 7fdcb3f0d700 1 --2- 172.21.15.139:0/1522819457 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdc9402ff00 0x7fdc940323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:31.765 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.775+0000 7fdcb3f0d700 1 --2- 172.21.15.139:0/1522819457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcac0fd2c0 0x7fdcac0fcb90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:31.765 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.775+0000 7fdcb3f0d700 1 -- 172.21.15.139:0/1522819457 >> 172.21.15.139:0/1522819457 conn(0x7fdcac0f5a80 msgr2=0x7fdcac0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:31.765 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.775+0000 7fdcb3f0d700 1 -- 172.21.15.139:0/1522819457 shutdown_connections 2021-11-20T05:59:31.766 INFO:teuthology.orchestra.run.smithi139.stderr:/usr/bin/ceph: 2021-11-20T05:59:31.775+0000 7fdcb3f0d700 1 -- 172.21.15.139:0/1522819457 wait complete. 2021-11-20T05:59:31.908 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:31 smithi139 conmon[33501]: audit 2021-11-20T05:59: 2021-11-20T05:59:31.908 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:31 smithi139 conmon[33501]: 31.774427+0000 mon.smithi139 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.139:0/1522819457' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2021-11-20T05:59:32.104 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-20T05:59:32.113 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: disabled 2021-11-20T05:59:32.113 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: systemctl is-active firewalld.service 2021-11-20T05:59:32.119 INFO:teuthology.orchestra.run.smithi139.stderr:systemctl: inactive 2021-11-20T05:59:32.120 INFO:teuthology.orchestra.run.smithi139.stderr:firewalld.service is not enabled 2021-11-20T05:59:32.120 INFO:teuthology.orchestra.run.smithi139.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2021-11-20T05:59:32.124 INFO:teuthology.orchestra.run.smithi139.stderr:Ceph Dashboard is now available at: 2021-11-20T05:59:32.124 INFO:teuthology.orchestra.run.smithi139.stderr: 2021-11-20T05:59:32.124 INFO:teuthology.orchestra.run.smithi139.stderr: URL: https://smithi139.front.sepia.ceph.com:8443/ 2021-11-20T05:59:32.124 INFO:teuthology.orchestra.run.smithi139.stderr: User: admin 2021-11-20T05:59:32.125 INFO:teuthology.orchestra.run.smithi139.stderr: Password: rzdz9yximn 2021-11-20T05:59:32.125 INFO:teuthology.orchestra.run.smithi139.stderr: 2021-11-20T05:59:32.126 INFO:teuthology.orchestra.run.smithi139.stderr:You can access the Ceph CLI with: 2021-11-20T05:59:32.126 INFO:teuthology.orchestra.run.smithi139.stderr: 2021-11-20T05:59:32.126 INFO:teuthology.orchestra.run.smithi139.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2021-11-20T05:59:32.127 INFO:teuthology.orchestra.run.smithi139.stderr: 2021-11-20T05:59:32.127 INFO:teuthology.orchestra.run.smithi139.stderr:Please consider enabling telemetry to help improve Ceph: 2021-11-20T05:59:32.127 INFO:teuthology.orchestra.run.smithi139.stderr: 2021-11-20T05:59:32.127 INFO:teuthology.orchestra.run.smithi139.stderr: ceph telemetry on 2021-11-20T05:59:32.128 INFO:teuthology.orchestra.run.smithi139.stderr: 2021-11-20T05:59:32.128 INFO:teuthology.orchestra.run.smithi139.stderr:For more information see: 2021-11-20T05:59:32.128 INFO:teuthology.orchestra.run.smithi139.stderr: 2021-11-20T05:59:32.128 INFO:teuthology.orchestra.run.smithi139.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2021-11-20T05:59:32.128 INFO:teuthology.orchestra.run.smithi139.stderr: 2021-11-20T05:59:32.129 INFO:teuthology.orchestra.run.smithi139.stderr:Bootstrap complete. 2021-11-20T05:59:32.129 INFO:teuthology.orchestra.run.smithi139.stderr:Releasing lock 140214527003336 on /run/cephadm/d819f2cc-49c6-11ec-8c2c-001a4aab830c.lock 2021-11-20T05:59:32.129 INFO:teuthology.orchestra.run.smithi139.stderr:Lock 140214527003336 released on /run/cephadm/d819f2cc-49c6-11ec-8c2c-001a4aab830c.lock 2021-11-20T05:59:32.158 INFO:tasks.cephadm:Fetching config... 2021-11-20T05:59:32.159 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-11-20T05:59:32.159 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2021-11-20T05:59:32.176 INFO:tasks.cephadm:Fetching client.admin keyring... 2021-11-20T05:59:32.177 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-11-20T05:59:32.177 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2021-11-20T05:59:32.231 INFO:tasks.cephadm:Fetching mon keyring... 2021-11-20T05:59:32.232 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-11-20T05:59:32.232 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/keyring of=/dev/stdout 2021-11-20T05:59:32.298 INFO:tasks.cephadm:Fetching pub ssh key... 2021-11-20T05:59:32.299 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-11-20T05:59:32.299 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2021-11-20T05:59:32.355 INFO:tasks.cephadm:Installing pub ssh key for root users... 2021-11-20T05:59:32.356 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDF3MRPQBm4E6xqSiStjeAOMS5Dh0rqP7cp+aNJcOCJ0Ssvwrw/sDmwga8nKkzL/87vY0tF9SKU1XftEPgEosvXbTKb/9sxFyPyQ5jPjnnI26NO7NR338Dt4vszFmtW2YN0NaqGWgGLU+0xcYIg768vZaEIMOIaWhiAcNPvxCqhjKuaOcNConKsSgDwjYjO/KYdFbN2iGTE90/onuktSOhM/NOSTIzqPyMGXIdiLStjG0rmGBTfJMKdLVY95xx8oa6Is6HDqWF6xxTYAMQ1mMZs2cy2O7UCh2kKiQQSH7yA741CLvUMBJiusnkSXR13r7jUd7Ml28QvhUzBm2VgKZrMWQXtRv9eH81qn+UXgCuBaXCmbZKTDdpnw6ok5cNM55/MA1Kq2Al7zEGA8vfZ1Cx0CWeJibUoG91Ho1nolTH8eSN9gNUeiuRF1Uk/WBSKiA6YHIqqdWTd92RhbwnnSA1m5qDECWq8CZZN5/wK3QarHsAl8swpoiM+r+1HTmvWclU= ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-11-20T05:59:32.432 INFO:teuthology.orchestra.run.smithi139.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDF3MRPQBm4E6xqSiStjeAOMS5Dh0rqP7cp+aNJcOCJ0Ssvwrw/sDmwga8nKkzL/87vY0tF9SKU1XftEPgEosvXbTKb/9sxFyPyQ5jPjnnI26NO7NR338Dt4vszFmtW2YN0NaqGWgGLU+0xcYIg768vZaEIMOIaWhiAcNPvxCqhjKuaOcNConKsSgDwjYjO/KYdFbN2iGTE90/onuktSOhM/NOSTIzqPyMGXIdiLStjG0rmGBTfJMKdLVY95xx8oa6Is6HDqWF6xxTYAMQ1mMZs2cy2O7UCh2kKiQQSH7yA741CLvUMBJiusnkSXR13r7jUd7Ml28QvhUzBm2VgKZrMWQXtRv9eH81qn+UXgCuBaXCmbZKTDdpnw6ok5cNM55/MA1Kq2Al7zEGA8vfZ1Cx0CWeJibUoG91Ho1nolTH8eSN9gNUeiuRF1Uk/WBSKiA6YHIqqdWTd92RhbwnnSA1m5qDECWq8CZZN5/wK3QarHsAl8swpoiM+r+1HTmvWclU= ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T05:59:32.443 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDF3MRPQBm4E6xqSiStjeAOMS5Dh0rqP7cp+aNJcOCJ0Ssvwrw/sDmwga8nKkzL/87vY0tF9SKU1XftEPgEosvXbTKb/9sxFyPyQ5jPjnnI26NO7NR338Dt4vszFmtW2YN0NaqGWgGLU+0xcYIg768vZaEIMOIaWhiAcNPvxCqhjKuaOcNConKsSgDwjYjO/KYdFbN2iGTE90/onuktSOhM/NOSTIzqPyMGXIdiLStjG0rmGBTfJMKdLVY95xx8oa6Is6HDqWF6xxTYAMQ1mMZs2cy2O7UCh2kKiQQSH7yA741CLvUMBJiusnkSXR13r7jUd7Ml28QvhUzBm2VgKZrMWQXtRv9eH81qn+UXgCuBaXCmbZKTDdpnw6ok5cNM55/MA1Kq2Al7zEGA8vfZ1Cx0CWeJibUoG91Ho1nolTH8eSN9gNUeiuRF1Uk/WBSKiA6YHIqqdWTd92RhbwnnSA1m5qDECWq8CZZN5/wK3QarHsAl8swpoiM+r+1HTmvWclU= ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-11-20T05:59:32.487 INFO:teuthology.orchestra.run.smithi178.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDF3MRPQBm4E6xqSiStjeAOMS5Dh0rqP7cp+aNJcOCJ0Ssvwrw/sDmwga8nKkzL/87vY0tF9SKU1XftEPgEosvXbTKb/9sxFyPyQ5jPjnnI26NO7NR338Dt4vszFmtW2YN0NaqGWgGLU+0xcYIg768vZaEIMOIaWhiAcNPvxCqhjKuaOcNConKsSgDwjYjO/KYdFbN2iGTE90/onuktSOhM/NOSTIzqPyMGXIdiLStjG0rmGBTfJMKdLVY95xx8oa6Is6HDqWF6xxTYAMQ1mMZs2cy2O7UCh2kKiQQSH7yA741CLvUMBJiusnkSXR13r7jUd7Ml28QvhUzBm2VgKZrMWQXtRv9eH81qn+UXgCuBaXCmbZKTDdpnw6ok5cNM55/MA1Kq2Al7zEGA8vfZ1Cx0CWeJibUoG91Ho1nolTH8eSN9gNUeiuRF1Uk/WBSKiA6YHIqqdWTd92RhbwnnSA1m5qDECWq8CZZN5/wK3QarHsAl8swpoiM+r+1HTmvWclU= ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T05:59:32.500 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2021-11-20T05:59:34.149 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:34 smithi139 conmon[33501]: cluster 2021-11-20T05:59:33.130612+0000 mon.smithi139 (mon.0) 112 : cluster [DBG] mgrmap e16: smithi139.dplyxn(active, since 7s) 2021-11-20T05:59:34.194 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.206+0000 7f9d5f0de700 1 -- 172.21.15.139:0/409716164 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9d580f51a0 msgr2=0x7f9d580f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:34.195 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.206+0000 7f9d5f0de700 1 --2- 172.21.15.139:0/409716164 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9d580f51a0 0x7f9d580f55c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f9d48009fb0 tx=0x7f9d48009b30).stop 2021-11-20T05:59:34.195 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.206+0000 7f9d5f0de700 1 -- 172.21.15.139:0/409716164 shutdown_connections 2021-11-20T05:59:34.195 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.206+0000 7f9d5f0de700 1 --2- 172.21.15.139:0/409716164 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9d580f51a0 0x7f9d580f55c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:34.195 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.206+0000 7f9d5f0de700 1 -- 172.21.15.139:0/409716164 >> 172.21.15.139:0/409716164 conn(0x7f9d580f09a0 msgr2=0x7f9d580f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:34.196 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.207+0000 7f9d5f0de700 1 -- 172.21.15.139:0/409716164 shutdown_connections 2021-11-20T05:59:34.196 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.207+0000 7f9d5f0de700 1 -- 172.21.15.139:0/409716164 wait complete. 2021-11-20T05:59:34.196 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.207+0000 7f9d5f0de700 1 Processor -- start 2021-11-20T05:59:34.196 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.207+0000 7f9d5f0de700 1 -- start start 2021-11-20T05:59:34.197 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.207+0000 7f9d5f0de700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9d580f51a0 0x7f9d5810b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:34.197 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.207+0000 7f9d5f0de700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d58106fe0 con 0x7f9d580f51a0 2021-11-20T05:59:34.197 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.207+0000 7f9d5e0dc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9d580f51a0 0x7f9d5810b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:34.197 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.207+0000 7f9d5e0dc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9d580f51a0 0x7f9d5810b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56764/0 (socket says 172.21.15.139:56764) 2021-11-20T05:59:34.198 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.207+0000 7f9d5e0dc700 1 -- 172.21.15.139:0/1281022172 learned_addr learned my addr 172.21.15.139:0/1281022172 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:34.198 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.208+0000 7f9d5e0dc700 1 -- 172.21.15.139:0/1281022172 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d48005040 con 0x7f9d580f51a0 2021-11-20T05:59:34.198 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.208+0000 7f9d5e0dc700 1 --2- 172.21.15.139:0/1281022172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9d580f51a0 0x7f9d5810b5f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9d48013510 tx=0x7f9d4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:34.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.208+0000 7f9d4effd700 1 -- 172.21.15.139:0/1281022172 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d48009320 con 0x7f9d580f51a0 2021-11-20T05:59:34.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.208+0000 7f9d4effd700 1 -- 172.21.15.139:0/1281022172 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d48013f00 con 0x7f9d580f51a0 2021-11-20T05:59:34.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.208+0000 7f9d4effd700 1 -- 172.21.15.139:0/1281022172 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d48014dc0 con 0x7f9d580f51a0 2021-11-20T05:59:34.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.208+0000 7f9d5f0de700 1 -- 172.21.15.139:0/1281022172 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d58060220 con 0x7f9d580f51a0 2021-11-20T05:59:34.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.208+0000 7f9d5f0de700 1 -- 172.21.15.139:0/1281022172 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d5810bae0 con 0x7f9d580f51a0 2021-11-20T05:59:34.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.209+0000 7f9d5f0de700 1 -- 172.21.15.139:0/1281022172 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d58045ad0 con 0x7f9d580f51a0 2021-11-20T05:59:34.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.209+0000 7f9d4effd700 1 -- 172.21.15.139:0/1281022172 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d480095f0 con 0x7f9d580f51a0 2021-11-20T05:59:34.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.209+0000 7f9d4effd700 1 --2- 172.21.15.139:0/1281022172 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9d4402ff60 0x7f9d44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:34.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.209+0000 7f9d4effd700 1 -- 172.21.15.139:0/1281022172 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d48041000 con 0x7f9d580f51a0 2021-11-20T05:59:34.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.209+0000 7f9d5d8db700 1 --2- 172.21.15.139:0/1281022172 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9d4402ff60 0x7f9d44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:34.203 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.210+0000 7f9d5d8db700 1 --2- 172.21.15.139:0/1281022172 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9d4402ff60 0x7f9d44032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9d540096a0 tx=0x7f9d54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:34.203 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.213+0000 7f9d4effd700 1 -- 172.21.15.139:0/1281022172 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d48014420 con 0x7f9d580f51a0 2021-11-20T05:59:34.363 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.377+0000 7f9d5f0de700 1 -- 172.21.15.139:0/1281022172 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f9d580fabe0 con 0x7f9d580f51a0 2021-11-20T05:59:34.365 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.380+0000 7f9d4effd700 1 -- 172.21.15.139:0/1281022172 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9d48010230 con 0x7f9d580f51a0 2021-11-20T05:59:34.369 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.381+0000 7f9d5f0de700 1 -- 172.21.15.139:0/1281022172 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9d4402ff60 msgr2=0x7f9d44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:34.369 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.381+0000 7f9d5f0de700 1 --2- 172.21.15.139:0/1281022172 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9d4402ff60 0x7f9d44032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9d540096a0 tx=0x7f9d54006b40).stop 2021-11-20T05:59:34.370 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.381+0000 7f9d5f0de700 1 -- 172.21.15.139:0/1281022172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9d580f51a0 msgr2=0x7f9d5810b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:34.370 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.381+0000 7f9d5f0de700 1 --2- 172.21.15.139:0/1281022172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9d580f51a0 0x7f9d5810b5f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9d48013510 tx=0x7f9d4800b040).stop 2021-11-20T05:59:34.370 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.381+0000 7f9d5f0de700 1 -- 172.21.15.139:0/1281022172 shutdown_connections 2021-11-20T05:59:34.371 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.381+0000 7f9d5f0de700 1 --2- 172.21.15.139:0/1281022172 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9d4402ff60 0x7f9d44032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:34.371 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.381+0000 7f9d5f0de700 1 --2- 172.21.15.139:0/1281022172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9d580f51a0 0x7f9d5810b5f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:34.371 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.381+0000 7f9d5f0de700 1 -- 172.21.15.139:0/1281022172 >> 172.21.15.139:0/1281022172 conn(0x7f9d580f09a0 msgr2=0x7f9d580f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:34.372 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.382+0000 7f9d5f0de700 1 -- 172.21.15.139:0/1281022172 shutdown_connections 2021-11-20T05:59:34.372 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:34.382+0000 7f9d5f0de700 1 -- 172.21.15.139:0/1281022172 wait complete. 2021-11-20T05:59:34.754 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi178 2021-11-20T05:59:34.754 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2021-11-20T05:59:34.755 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.conf 2021-11-20T05:59:34.774 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2021-11-20T05:59:34.774 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.client.admin.keyring 2021-11-20T05:59:34.832 INFO:tasks.cephadm:Adding host smithi178 to orchestrator... 2021-11-20T05:59:34.833 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph orch host add smithi178 2021-11-20T05:59:35.630 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:35 smithi139 conmon[33501]: audit 2021-11-20T05:59:34.380545+0000 mon.smithi139 (mon.0) 113 : audit [INF] from='client.? 172.21.15.139:0/1281022172' entity='client.admin' 2021-11-20T05:59:38.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:38 smithi139 conmon[33501]: audit 2021-11-20T05:59:37.117680+0000 mon.smithi139 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:38.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:38 smithi139 conmon[33501]: audit 2021-11-20T05:59:37.478768+0000 mon.smithi139 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:39.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:39 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:37.484796+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T05:59:39.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:39 smithi139 conmon[33501]: 14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi139 on smithi139 2021-11-20T05:59:44.144 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:43 smithi139 conmon[33501]: audit 2021-11-20T05:59:42.884564+0000 mon.smithi139 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:44.144 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:43 smithi139 conmon[33501]: audit 2021-11-20T05:59:42.885550+0000 mon.smithi139 (mon.0) 117 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi139", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T05:59:44.145 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:43 smithi139 conmon[33501]: audit 2021-11-20T05:59:42.887989+0000 mon.smithi139 (mon.0) 118 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi139", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2021-11-20T05:59:44.145 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:43 smithi139 conmon[33501]: audit 2021-11-20T05:59:42.888652+0000 mon.smithi139 (mon.0) 119 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T05:59:44.145 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:43 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:42.889178+0000 mgr.smithi139.dplyxn (mgr.14162) 6 : cephadm [INF] Deploying daemon crash.smithi139 on smithi139 2021-11-20T05:59:46.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:45 smithi139 conmon[33501]: audit 2021-11-20T05:59:44.936432+0000 mon.smithi139 (mon.0) 120 : 2021-11-20T05:59:46.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:45 smithi139 conmon[33501]: audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:46.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:45 smithi139 conmon[33501]: audit 2021-11-20T05:59:45.015979+0000 mon.smithi139 (mon.0) 121 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:46.280 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:45 smithi139 conmon[33501]: audit 2021-11-20T05:59:45.018544+0000 mon.smithi139 (mon.0) 122 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:46.280 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:45 smithi139 conmon[33501]: audit 2021-11-20T05:59:45.018930+0000 mon.smithi139 (mon.0) 123 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-11-20T05:59:46.280 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:45 smithi139 conmon[33501]: audit 2021-11-20T05:59:45.019378+0000 mgr.smithi139.dplyxn (mgr.14162) 7 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-11-20T05:59:46.281 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:45 smithi139 conmon[33501]: audit 2021-11-20T05:59:45.022434+0000 mon.smithi139 (mon.0) 124 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:46.282 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:45 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:45.025314+0000 mgr.smithi139.dplyxn (mgr.14162) 8 : cephadm [INF] Deploying daemon grafana.smithi139 on smithi139 2021-11-20T05:59:48.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:48 smithi139 conmon[33501]: cluster 2021-11-20T05:59:46.172576+0000 2021-11-20T05:59:48.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:48 smithi139 conmon[33501]: mgr.smithi139.dplyxn (mgr.14162) 9 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:59:50.041 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.051+0000 7f392893b700 1 -- 172.21.15.178:0/1986368627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39240f51a0 msgr2=0x7f39240f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:50.042 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.051+0000 7f392893b700 1 --2- 172.21.15.178:0/1986368627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39240f51a0 0x7f39240f55c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f3914009fb0 tx=0x7f3914009b30).stop 2021-11-20T05:59:50.042 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.051+0000 7f392893b700 1 -- 172.21.15.178:0/1986368627 shutdown_connections 2021-11-20T05:59:50.042 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.051+0000 7f392893b700 1 --2- 172.21.15.178:0/1986368627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39240f51a0 0x7f39240f55c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:50.042 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.051+0000 7f392893b700 1 -- 172.21.15.178:0/1986368627 >> 172.21.15.178:0/1986368627 conn(0x7f39240f09a0 msgr2=0x7f39240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:50.043 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.051+0000 7f392893b700 1 -- 172.21.15.178:0/1986368627 shutdown_connections 2021-11-20T05:59:50.044 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.051+0000 7f392893b700 1 -- 172.21.15.178:0/1986368627 wait complete. 2021-11-20T05:59:50.044 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.052+0000 7f392893b700 1 Processor -- start 2021-11-20T05:59:50.044 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.052+0000 7f392893b700 1 -- start start 2021-11-20T05:59:50.044 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.052+0000 7f392893b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39240f51a0 0x7f39241053d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:50.044 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.052+0000 7f392893b700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39241058c0 con 0x7f39240f51a0 2021-11-20T05:59:50.045 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.052+0000 7f39237fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39240f51a0 0x7f39241053d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:50.045 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.052+0000 7f39237fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39240f51a0 0x7f39241053d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.178:33418/0 (socket says 172.21.15.178:33418) 2021-11-20T05:59:50.045 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.052+0000 7f39237fe700 1 -- 172.21.15.178:0/923959746 learned_addr learned my addr 172.21.15.178:0/923959746 (peer_addr_for_me v2:172.21.15.178:0/0) 2021-11-20T05:59:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.053+0000 7f39237fe700 1 -- 172.21.15.178:0/923959746 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3914005040 con 0x7f39240f51a0 2021-11-20T05:59:50.046 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.053+0000 7f39237fe700 1 --2- 172.21.15.178:0/923959746 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39240f51a0 0x7f39241053d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f3914013510 tx=0x7f391400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:50.047 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.053+0000 7f390bfff700 1 -- 172.21.15.178:0/923959746 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3914009320 con 0x7f39240f51a0 2021-11-20T05:59:50.047 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.053+0000 7f392893b700 1 -- 172.21.15.178:0/923959746 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3924105ac0 con 0x7f39240f51a0 2021-11-20T05:59:50.047 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.053+0000 7f390bfff700 1 -- 172.21.15.178:0/923959746 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3914013f00 con 0x7f39240f51a0 2021-11-20T05:59:50.048 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.053+0000 7f390bfff700 1 -- 172.21.15.178:0/923959746 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3914014dc0 con 0x7f39240f51a0 2021-11-20T05:59:50.048 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.053+0000 7f392893b700 1 -- 172.21.15.178:0/923959746 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3924105f50 con 0x7f39240f51a0 2021-11-20T05:59:50.048 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.054+0000 7f390bfff700 1 -- 172.21.15.178:0/923959746 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f391401d3d0 con 0x7f39240f51a0 2021-11-20T05:59:50.049 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.054+0000 7f392893b700 1 -- 172.21.15.178:0/923959746 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3924045ad0 con 0x7f39240f51a0 2021-11-20T05:59:50.049 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.054+0000 7f390bfff700 1 --2- 172.21.15.178:0/923959746 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f390c02ff60 0x7f390c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:50.049 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.054+0000 7f390bfff700 1 -- 172.21.15.178:0/923959746 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39140413f0 con 0x7f39240f51a0 2021-11-20T05:59:50.049 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.054+0000 7f3922ffd700 1 --2- 172.21.15.178:0/923959746 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f390c02ff60 0x7f390c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:50.050 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.055+0000 7f3922ffd700 1 --2- 172.21.15.178:0/923959746 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f390c02ff60 0x7f390c032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f39180096a0 tx=0x7f3918006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:50.050 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.060+0000 7f390bfff700 1 -- 172.21.15.178:0/923959746 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f391401cb10 con 0x7f39240f51a0 2021-11-20T05:59:50.218 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.231+0000 7f392893b700 1 -- 172.21.15.178:0/923959746 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}) v1 -- 0x7f39240f1b30 con 0x7f390c02ff60 2021-11-20T05:59:50.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:50 smithi139 conmon[33501]: cluster 2021-11-20T05:59:48.172783+0000 mgr.smithi139.dplyxn (mgr.14162) 10 : cluster 2021-11-20T05:59:50.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:50 smithi139 conmon[33501]: [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:59:50.897 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.908+0000 7f390bfff700 1 -- 172.21.15.178:0/923959746 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f39240f1b30 con 0x7f390c02ff60 2021-11-20T05:59:50.898 INFO:teuthology.orchestra.run.smithi178.stdout:Added host 'smithi178' 2021-11-20T05:59:50.899 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.911+0000 7f392893b700 1 -- 172.21.15.178:0/923959746 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f390c02ff60 msgr2=0x7f390c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:50.899 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.911+0000 7f392893b700 1 --2- 172.21.15.178:0/923959746 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f390c02ff60 0x7f390c032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f39180096a0 tx=0x7f3918006b40).stop 2021-11-20T05:59:50.900 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.911+0000 7f392893b700 1 -- 172.21.15.178:0/923959746 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39240f51a0 msgr2=0x7f39241053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:50.900 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.911+0000 7f392893b700 1 --2- 172.21.15.178:0/923959746 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39240f51a0 0x7f39241053d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f3914013510 tx=0x7f391400b040).stop 2021-11-20T05:59:50.900 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.911+0000 7f392893b700 1 -- 172.21.15.178:0/923959746 shutdown_connections 2021-11-20T05:59:50.900 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.911+0000 7f392893b700 1 --2- 172.21.15.178:0/923959746 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f390c02ff60 0x7f390c032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:50.900 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.911+0000 7f392893b700 1 --2- 172.21.15.178:0/923959746 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39240f51a0 0x7f39241053d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:50.901 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.911+0000 7f392893b700 1 -- 172.21.15.178:0/923959746 >> 172.21.15.178:0/923959746 conn(0x7f39240f09a0 msgr2=0x7f39240f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:50.901 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.911+0000 7f392893b700 1 -- 172.21.15.178:0/923959746 shutdown_connections 2021-11-20T05:59:50.901 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:50.911+0000 7f392893b700 1 -- 172.21.15.178:0/923959746 wait complete. 2021-11-20T05:59:51.401 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph orch host ls --format=json 2021-11-20T05:59:52.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:51 smithi139 conmon[33501]: cluster 2021-11-20T05:59:50.173044+0000 mgr.smithi139.dplyxn (mgr.14162) 11 2021-11-20T05:59:52.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:51 smithi139 conmon[33501]: : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:59:52.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:51 smithi139 conmon[33501]: audit 2021-11-20T05:59:50.233577+0000 mgr.smithi139.dplyxn (mgr.14162) 12 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:59:52.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:51 smithi139 conmon[33501]: audit 2021-11-20T05:59:50.909477+0000 mon.smithi139 (mon.0) 125 : audit 2021-11-20T05:59:52.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:51 smithi139 conmon[33501]: [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:52.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:51 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:50.909906+0000 mgr.smithi139.dplyxn (mgr.14162) 13 : cephadm [INF] 2021-11-20T05:59:52.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:51 smithi139 conmon[33501]: Added host smithi178 2021-11-20T05:59:52.925 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.936+0000 7f23be32a700 1 -- 172.21.15.178:0/3069242239 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23b80f5410 msgr2=0x7f23b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:52.926 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.936+0000 7f23be32a700 1 --2- 172.21.15.178:0/3069242239 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23b80f5410 0x7f23b80f5830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f23a8009fb0 tx=0x7f23a8009b30).stop 2021-11-20T05:59:52.926 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.936+0000 7f23be32a700 1 -- 172.21.15.178:0/3069242239 shutdown_connections 2021-11-20T05:59:52.926 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.936+0000 7f23be32a700 1 --2- 172.21.15.178:0/3069242239 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23b80f5410 0x7f23b80f5830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:52.927 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.936+0000 7f23be32a700 1 -- 172.21.15.178:0/3069242239 >> 172.21.15.178:0/3069242239 conn(0x7f23b80f0b70 msgr2=0x7f23b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:52.927 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.936+0000 7f23be32a700 1 -- 172.21.15.178:0/3069242239 shutdown_connections 2021-11-20T05:59:52.927 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.936+0000 7f23be32a700 1 -- 172.21.15.178:0/3069242239 wait complete. 2021-11-20T05:59:52.927 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.937+0000 7f23be32a700 1 Processor -- start 2021-11-20T05:59:52.927 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.937+0000 7f23be32a700 1 -- start start 2021-11-20T05:59:52.928 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.937+0000 7f23be32a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23b80f5410 0x7f23b810b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:52.928 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.937+0000 7f23be32a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b8107210 con 0x7f23b80f5410 2021-11-20T05:59:52.928 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.937+0000 7f23b7fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23b80f5410 0x7f23b810b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:52.928 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.937+0000 7f23b7fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23b80f5410 0x7f23b810b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.178:33424/0 (socket says 172.21.15.178:33424) 2021-11-20T05:59:52.929 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.937+0000 7f23b7fff700 1 -- 172.21.15.178:0/1775832218 learned_addr learned my addr 172.21.15.178:0/1775832218 (peer_addr_for_me v2:172.21.15.178:0/0) 2021-11-20T05:59:52.929 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.938+0000 7f23b7fff700 1 -- 172.21.15.178:0/1775832218 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a8005040 con 0x7f23b80f5410 2021-11-20T05:59:52.929 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.938+0000 7f23b7fff700 1 --2- 172.21.15.178:0/1775832218 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23b80f5410 0x7f23b810b820 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f23a8014000 tx=0x7f23a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:52.929 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.938+0000 7f23b4ff9700 1 -- 172.21.15.178:0/1775832218 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23a80095f0 con 0x7f23b80f5410 2021-11-20T05:59:52.929 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.938+0000 7f23be32a700 1 -- 172.21.15.178:0/1775832218 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b8060220 con 0x7f23b80f5410 2021-11-20T05:59:52.930 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.938+0000 7f23b4ff9700 1 -- 172.21.15.178:0/1775832218 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23a8007e70 con 0x7f23b80f5410 2021-11-20T05:59:52.930 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.938+0000 7f23b4ff9700 1 -- 172.21.15.178:0/1775832218 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23a8014ea0 con 0x7f23b80f5410 2021-11-20T05:59:52.930 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.938+0000 7f23be32a700 1 -- 172.21.15.178:0/1775832218 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b810bd10 con 0x7f23b80f5410 2021-11-20T05:59:52.931 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.939+0000 7f23b4ff9700 1 -- 172.21.15.178:0/1775832218 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23a8025070 con 0x7f23b80f5410 2021-11-20T05:59:52.932 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.939+0000 7f23b4ff9700 1 --2- 172.21.15.178:0/1775832218 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f23a002ffb0 0x7f23a0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:52.932 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.939+0000 7f23b4ff9700 1 -- 172.21.15.178:0/1775832218 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f23a8040a40 con 0x7f23b80f5410 2021-11-20T05:59:52.932 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.939+0000 7f23be32a700 1 -- 172.21.15.178:0/1775832218 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b8045ad0 con 0x7f23b80f5410 2021-11-20T05:59:52.932 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.939+0000 7f23b77fe700 1 --2- 172.21.15.178:0/1775832218 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f23a002ffb0 0x7f23a0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:52.933 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.939+0000 7f23b77fe700 1 --2- 172.21.15.178:0/1775832218 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f23a002ffb0 0x7f23a0032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f23ac0096a0 tx=0x7f23ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:52.933 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:52.944+0000 7f23b4ff9700 1 -- 172.21.15.178:0/1775832218 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23a8026350 con 0x7f23b80f5410 2021-11-20T05:59:53.105 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:53.117+0000 7f23be32a700 1 -- 172.21.15.178:0/1775832218 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f23b80f1d40 con 0x7f23a002ffb0 2021-11-20T05:59:53.106 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:53.119+0000 7f23b4ff9700 1 -- 172.21.15.178:0/1775832218 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+157 (secure 0 0 0) 0x7f23b80f1d40 con 0x7f23a002ffb0 2021-11-20T05:59:53.107 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T05:59:53.107 INFO:teuthology.orchestra.run.smithi178.stdout:[{"addr": "172.21.15.139", "hostname": "smithi139", "labels": [], "status": ""}, {"addr": "smithi178", "hostname": "smithi178", "labels": [], "status": ""}] 2021-11-20T05:59:53.108 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:53.120+0000 7f23be32a700 1 -- 172.21.15.178:0/1775832218 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f23a002ffb0 msgr2=0x7f23a0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:53.108 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:53.120+0000 7f23be32a700 1 --2- 172.21.15.178:0/1775832218 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f23a002ffb0 0x7f23a0032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f23ac0096a0 tx=0x7f23ac006b40).stop 2021-11-20T05:59:53.109 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:53.120+0000 7f23be32a700 1 -- 172.21.15.178:0/1775832218 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23b80f5410 msgr2=0x7f23b810b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:53.109 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:53.120+0000 7f23be32a700 1 --2- 172.21.15.178:0/1775832218 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23b80f5410 0x7f23b810b820 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f23a8014000 tx=0x7f23a800b040).stop 2021-11-20T05:59:53.109 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:53.120+0000 7f23be32a700 1 -- 172.21.15.178:0/1775832218 shutdown_connections 2021-11-20T05:59:53.109 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:53.120+0000 7f23be32a700 1 --2- 172.21.15.178:0/1775832218 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f23a002ffb0 0x7f23a0032470 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:53.109 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:53.120+0000 7f23be32a700 1 --2- 172.21.15.178:0/1775832218 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23b80f5410 0x7f23b810b820 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:53.110 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:53.120+0000 7f23be32a700 1 -- 172.21.15.178:0/1775832218 >> 172.21.15.178:0/1775832218 conn(0x7f23b80f0b70 msgr2=0x7f23b80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:53.110 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:53.121+0000 7f23be32a700 1 -- 172.21.15.178:0/1775832218 shutdown_connections 2021-11-20T05:59:53.110 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:53.121+0000 7f23be32a700 1 -- 172.21.15.178:0/1775832218 wait complete. 2021-11-20T05:59:53.532 INFO:tasks.cephadm:Setting crush tunables to default 2021-11-20T05:59:53.533 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph osd crush tunables default 2021-11-20T05:59:54.402 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:54 smithi139 conmon[33501]: cluster 2021-11-20T05:59:52.173306+0000 mgr.smithi139.dplyxn (mgr.14162) 14 : cluster [DBG] 2021-11-20T05:59:54.402 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:54 smithi139 conmon[33501]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:59:54.403 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:54 smithi139 conmon[33501]: audit 2021-11-20T05:59:53.120342+0000 mgr.smithi139.dplyxn (mgr.14162) 15 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T05:59:55.411 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.420+0000 7f4c19754700 1 -- 172.21.15.139:0/741887760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c140f6510 msgr2=0x7f4c140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:55.411 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.420+0000 7f4c19754700 1 --2- 172.21.15.139:0/741887760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c140f6510 0x7f4c140f6930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f4c08009fb0 tx=0x7f4c08009b30).stop 2021-11-20T05:59:55.411 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.420+0000 7f4c19754700 1 -- 172.21.15.139:0/741887760 shutdown_connections 2021-11-20T05:59:55.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.420+0000 7f4c19754700 1 --2- 172.21.15.139:0/741887760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c140f6510 0x7f4c140f6930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:55.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.420+0000 7f4c19754700 1 -- 172.21.15.139:0/741887760 >> 172.21.15.139:0/741887760 conn(0x7f4c140f1c70 msgr2=0x7f4c140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:55.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.420+0000 7f4c19754700 1 -- 172.21.15.139:0/741887760 shutdown_connections 2021-11-20T05:59:55.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.420+0000 7f4c19754700 1 -- 172.21.15.139:0/741887760 wait complete. 2021-11-20T05:59:55.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.421+0000 7f4c19754700 1 Processor -- start 2021-11-20T05:59:55.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.421+0000 7f4c19754700 1 -- start start 2021-11-20T05:59:55.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.421+0000 7f4c19754700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c140f6510 0x7f4c141088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:55.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.421+0000 7f4c19754700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c14108de0 con 0x7f4c140f6510 2021-11-20T05:59:55.414 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.421+0000 7f4c12ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c140f6510 0x7f4c141088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:55.414 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.421+0000 7f4c12ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c140f6510 0x7f4c141088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56802/0 (socket says 172.21.15.139:56802) 2021-11-20T05:59:55.414 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.421+0000 7f4c12ffd700 1 -- 172.21.15.139:0/749207609 learned_addr learned my addr 172.21.15.139:0/749207609 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T05:59:55.414 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.421+0000 7f4c12ffd700 1 -- 172.21.15.139:0/749207609 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c08005040 con 0x7f4c140f6510 2021-11-20T05:59:55.415 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.422+0000 7f4c12ffd700 1 --2- 172.21.15.139:0/749207609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c140f6510 0x7f4c141088f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f4c08013510 tx=0x7f4c0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:55.415 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.422+0000 7f4bfb7fe700 1 -- 172.21.15.139:0/749207609 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c08009320 con 0x7f4c140f6510 2021-11-20T05:59:55.416 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.422+0000 7f4c19754700 1 -- 172.21.15.139:0/749207609 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c14108fe0 con 0x7f4c140f6510 2021-11-20T05:59:55.416 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.422+0000 7f4bfb7fe700 1 -- 172.21.15.139:0/749207609 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c08013f00 con 0x7f4c140f6510 2021-11-20T05:59:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.422+0000 7f4bfb7fe700 1 -- 172.21.15.139:0/749207609 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c08014d90 con 0x7f4c140f6510 2021-11-20T05:59:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.422+0000 7f4c19754700 1 -- 172.21.15.139:0/749207609 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c14109470 con 0x7f4c140f6510 2021-11-20T05:59:55.417 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.423+0000 7f4bfb7fe700 1 -- 172.21.15.139:0/749207609 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c080263a0 con 0x7f4c140f6510 2021-11-20T05:59:55.418 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.423+0000 7f4bfb7fe700 1 --2- 172.21.15.139:0/749207609 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4bfc02ffb0 0x7f4bfc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:55.418 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.423+0000 7f4bfb7fe700 1 -- 172.21.15.139:0/749207609 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c08040fe0 con 0x7f4c140f6510 2021-11-20T05:59:55.418 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.423+0000 7f4c19754700 1 -- 172.21.15.139:0/749207609 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c14045ad0 con 0x7f4c140f6510 2021-11-20T05:59:55.419 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.423+0000 7f4c127fc700 1 --2- 172.21.15.139:0/749207609 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4bfc02ffb0 0x7f4bfc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:55.420 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.423+0000 7f4c127fc700 1 --2- 172.21.15.139:0/749207609 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4bfc02ffb0 0x7f4bfc032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f4c040096a0 tx=0x7f4c04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:55.420 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.428+0000 7f4bfb7fe700 1 -- 172.21.15.139:0/749207609 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c08012920 con 0x7f4c140f6510 2021-11-20T05:59:55.586 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:55.599+0000 7f4c19754700 1 -- 172.21.15.139:0/749207609 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4c1410a190 con 0x7f4c140f6510 2021-11-20T05:59:56.127 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:56.135+0000 7f4bfb7fe700 1 -- 172.21.15.139:0/749207609 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4c0802b030 con 0x7f4c140f6510 2021-11-20T05:59:56.127 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:56.137+0000 7f4c19754700 1 -- 172.21.15.139:0/749207609 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4bfc02ffb0 msgr2=0x7f4bfc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:56.127 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:56.137+0000 7f4c19754700 1 --2- 172.21.15.139:0/749207609 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4bfc02ffb0 0x7f4bfc032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f4c040096a0 tx=0x7f4c04006b40).stop 2021-11-20T05:59:56.127 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:56.137+0000 7f4c19754700 1 -- 172.21.15.139:0/749207609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c140f6510 msgr2=0x7f4c141088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:56.128 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:56.137+0000 7f4c19754700 1 --2- 172.21.15.139:0/749207609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c140f6510 0x7f4c141088f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f4c08013510 tx=0x7f4c0800b040).stop 2021-11-20T05:59:56.128 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:56.137+0000 7f4c19754700 1 -- 172.21.15.139:0/749207609 shutdown_connections 2021-11-20T05:59:56.128 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:56.137+0000 7f4c19754700 1 --2- 172.21.15.139:0/749207609 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4bfc02ffb0 0x7f4bfc032470 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:56.128 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:56.137+0000 7f4c19754700 1 --2- 172.21.15.139:0/749207609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c140f6510 0x7f4c141088f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:56.128 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:56.137+0000 7f4c19754700 1 -- 172.21.15.139:0/749207609 >> 172.21.15.139:0/749207609 conn(0x7f4c140f1c70 msgr2=0x7f4c140f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:56.129 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:56.137+0000 7f4c19754700 1 -- 172.21.15.139:0/749207609 shutdown_connections 2021-11-20T05:59:56.129 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T05:59:56.137+0000 7f4c19754700 1 -- 172.21.15.139:0/749207609 wait complete. 2021-11-20T05:59:56.129 INFO:teuthology.orchestra.run.smithi139.stderr:adjusted tunables profile to default 2021-11-20T05:59:56.451 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:56 smithi139 conmon[33501]: cluster 2021-11-20T05:59:54.173589+0000 mgr.smithi139.dplyxn (mgr.14162) 16 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:59:56.451 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:56 smithi139 conmon[33501]: audit 2021-11-20T05:59:55.601246+0000 mon.smithi139 (mon.0) 126 : audit [INF] from='client.? 172.21.15.139:0/749207609' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2021-11-20T05:59:56.452 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:56 smithi139 conmon[33501]: audit 2021-11-20T05:59:55.948697+0000 mon.smithi139 (mon.0) 127 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:56.453 INFO:tasks.cephadm:Adding mon.smithi139 on smithi139 2021-11-20T05:59:56.453 INFO:tasks.cephadm:Adding mon.smithi178 on smithi178 2021-11-20T05:59:56.454 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph orch apply mon '2;smithi139:172.21.15.139=smithi139;smithi178:172.21.15.178=smithi178' 2021-11-20T05:59:57.378 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:57 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:55.949962+0000 mgr.smithi139.dplyxn ( 2021-11-20T05:59:57.379 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:57 smithi139 conmon[33501]: mgr.14162) 17 : cephadm [INF] Deploying daemon node-exporter.smithi139 on smithi139 2021-11-20T05:59:57.379 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:57 smithi139 conmon[33501]: audit 2021-11-20T05:59:56.136922+0000 mon.smithi139 (mon.0) 128 : audit [INF] from='client.? 172.21.15.139:0/749207609' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2021-11-20T05:59:57.379 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:57 smithi139 conmon[33501]: cluster 2021-11-20T05:59:56.136971+0000 mon.smithi139 (mon.0) 129 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2021-11-20T05:59:58.018 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.029+0000 7f75427ac700 1 -- 172.21.15.178:0/901469177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f753c0f6510 msgr2=0x7f753c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:58.019 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.029+0000 7f75427ac700 1 --2- 172.21.15.178:0/901469177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f753c0f6510 0x7f753c0f6930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f752c009fb0 tx=0x7f752c009b30).stop 2021-11-20T05:59:58.019 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.029+0000 7f75427ac700 1 -- 172.21.15.178:0/901469177 shutdown_connections 2021-11-20T05:59:58.019 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.029+0000 7f75427ac700 1 --2- 172.21.15.178:0/901469177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f753c0f6510 0x7f753c0f6930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:58.019 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.029+0000 7f75427ac700 1 -- 172.21.15.178:0/901469177 >> 172.21.15.178:0/901469177 conn(0x7f753c0f1c70 msgr2=0x7f753c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:58.020 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.029+0000 7f75427ac700 1 -- 172.21.15.178:0/901469177 shutdown_connections 2021-11-20T05:59:58.020 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.029+0000 7f75427ac700 1 -- 172.21.15.178:0/901469177 wait complete. 2021-11-20T05:59:58.020 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.029+0000 7f75427ac700 1 Processor -- start 2021-11-20T05:59:58.020 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.030+0000 7f75427ac700 1 -- start start 2021-11-20T05:59:58.021 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.030+0000 7f75427ac700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f753c0f6510 0x7f753c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:58.021 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.030+0000 7f75427ac700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753c069e20 con 0x7f753c0f6510 2021-11-20T05:59:58.021 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.030+0000 7f753bfff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f753c0f6510 0x7f753c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:58.021 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.030+0000 7f753bfff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f753c0f6510 0x7f753c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.178:33430/0 (socket says 172.21.15.178:33430) 2021-11-20T05:59:58.022 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.030+0000 7f753bfff700 1 -- 172.21.15.178:0/3870901633 learned_addr learned my addr 172.21.15.178:0/3870901633 (peer_addr_for_me v2:172.21.15.178:0/0) 2021-11-20T05:59:58.022 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.030+0000 7f753bfff700 1 -- 172.21.15.178:0/3870901633 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c005040 con 0x7f753c0f6510 2021-11-20T05:59:58.022 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.030+0000 7f753bfff700 1 --2- 172.21.15.178:0/3870901633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f753c0f6510 0x7f753c06b9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f752c013510 tx=0x7f752c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:58.022 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.030+0000 7f75397fa700 1 -- 172.21.15.178:0/3870901633 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f752c009320 con 0x7f753c0f6510 2021-11-20T05:59:58.023 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.030+0000 7f75427ac700 1 -- 172.21.15.178:0/3870901633 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f753c069fc0 con 0x7f753c0f6510 2021-11-20T05:59:58.023 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.030+0000 7f75397fa700 1 -- 172.21.15.178:0/3870901633 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f752c013f00 con 0x7f753c0f6510 2021-11-20T05:59:58.023 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.030+0000 7f75397fa700 1 -- 172.21.15.178:0/3870901633 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f752c014d90 con 0x7f753c0f6510 2021-11-20T05:59:58.023 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.030+0000 7f75427ac700 1 -- 172.21.15.178:0/3870901633 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f753c06a450 con 0x7f753c0f6510 2021-11-20T05:59:58.023 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.031+0000 7f75397fa700 1 -- 172.21.15.178:0/3870901633 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f752c0263a0 con 0x7f753c0f6510 2021-11-20T05:59:58.024 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.031+0000 7f75427ac700 1 -- 172.21.15.178:0/3870901633 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f753c059100 con 0x7f753c0f6510 2021-11-20T05:59:58.024 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.031+0000 7f75397fa700 1 --2- 172.21.15.178:0/3870901633 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f752802ff60 0x7f7528032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T05:59:58.024 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.031+0000 7f75397fa700 1 -- 172.21.15.178:0/3870901633 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f752c0410e0 con 0x7f753c0f6510 2021-11-20T05:59:58.024 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.031+0000 7f7533fff700 1 --2- 172.21.15.178:0/3870901633 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f752802ff60 0x7f7528032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T05:59:58.025 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.032+0000 7f7533fff700 1 --2- 172.21.15.178:0/3870901633 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f752802ff60 0x7f7528032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f75240096a0 tx=0x7f7524006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T05:59:58.026 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.036+0000 7f75397fa700 1 -- 172.21.15.178:0/3870901633 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f752c012510 con 0x7f753c0f6510 2021-11-20T05:59:58.195 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.209+0000 7f75427ac700 1 -- 172.21.15.178:0/3870901633 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi139:172.21.15.139=smithi139;smithi178:172.21.15.178=smithi178", "target": ["mon-mgr", ""]}) v1 -- 0x7f753c0f2e70 con 0x7f752802ff60 2021-11-20T05:59:58.200 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.214+0000 7f75397fa700 1 -- 172.21.15.178:0/3870901633 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f753c0f2e70 con 0x7f752802ff60 2021-11-20T05:59:58.201 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mon update... 2021-11-20T05:59:58.202 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.215+0000 7f75427ac700 1 -- 172.21.15.178:0/3870901633 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f752802ff60 msgr2=0x7f7528032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:58.203 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.215+0000 7f75427ac700 1 --2- 172.21.15.178:0/3870901633 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f752802ff60 0x7f7528032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f75240096a0 tx=0x7f7524006b40).stop 2021-11-20T05:59:58.203 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.215+0000 7f75427ac700 1 -- 172.21.15.178:0/3870901633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f753c0f6510 msgr2=0x7f753c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T05:59:58.203 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.215+0000 7f75427ac700 1 --2- 172.21.15.178:0/3870901633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f753c0f6510 0x7f753c06b9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f752c013510 tx=0x7f752c00b040).stop 2021-11-20T05:59:58.203 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.215+0000 7f75427ac700 1 -- 172.21.15.178:0/3870901633 shutdown_connections 2021-11-20T05:59:58.204 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.215+0000 7f75427ac700 1 --2- 172.21.15.178:0/3870901633 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f752802ff60 0x7f7528032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:58.204 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.215+0000 7f75427ac700 1 --2- 172.21.15.178:0/3870901633 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f753c0f6510 0x7f753c06b9a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T05:59:58.204 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.215+0000 7f75427ac700 1 -- 172.21.15.178:0/3870901633 >> 172.21.15.178:0/3870901633 conn(0x7f753c0f1c70 msgr2=0x7f753c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T05:59:58.204 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.215+0000 7f75427ac700 1 -- 172.21.15.178:0/3870901633 shutdown_connections 2021-11-20T05:59:58.205 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T05:59:58.215+0000 7f75427ac700 1 -- 172.21.15.178:0/3870901633 wait complete. 2021-11-20T05:59:58.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:58 smithi139 conmon[33501]: cluster 2021-11-20T05:59:56.173942+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T05:59:58.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:58 smithi139 conmon[33501]: 18 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:59:58.707 DEBUG:teuthology.orchestra.run.smithi178:mon.smithi178> sudo journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi178.service 2021-11-20T05:59:58.710 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T05:59:58.710 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T05:59:58.736 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:-- Logs begin at Sat 2021-11-20 05:48:12 UTC. -- 2021-11-20T05:59:59.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:59 smithi139 conmon[33501]: cluster 2021-11-20T05:59:58.174319+0000 mgr.smithi139.dplyxn (mgr.14162) 19 : 2021-11-20T05:59:59.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:59 smithi139 conmon[33501]: cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T05:59:59.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:59 smithi139 conmon[33501]: audit 2021-11-20T05:59:58.210638+0000 mgr.smithi139.dplyxn (mgr.14162) 20 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi139:172.21.15.139=smithi139;smithi178:172.21.15.178=smithi178", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T05:59:59.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:59 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:58.211811+0000 mgr.smithi139.dplyxn (mgr.14162) 21 : cephadm [INF] Saving service mon spec with placement smithi139:172.21.15.139=smithi139;smithi178:172.21.15.178=smithi178;count:2 2021-11-20T05:59:59.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:59 smithi139 conmon[33501]: audit 2021-11-20T05:59:58.214530+0000 mon.smithi139 (mon.0) 130 : 2021-11-20T05:59:59.280 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:59 smithi139 conmon[33501]: audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:59.280 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:59 smithi139 conmon[33501]: audit 2021-11-20T05:59:59.104531+0000 mon.smithi139 (mon.0) 131 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T05:59:59.280 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:59 smithi139 conmon[33501]: cephadm 2021-11-20T05:59:59.109759+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T05:59:59.280 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 05:59:59 smithi139 conmon[33501]: ) 22 : cephadm [INF] Deploying daemon prometheus.smithi139 on smithi139 2021-11-20T06:00:00.291 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.303+0000 7f4aa01a8700 1 -- 172.21.15.178:0/4282793292 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a98060220 msgr2=0x7f4a980f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:00.292 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.303+0000 7f4aa01a8700 1 --2- 172.21.15.178:0/4282793292 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a98060220 0x7f4a980f9150 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f4a88009960 tx=0x7f4a88004d10).stop 2021-11-20T06:00:00.292 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.303+0000 7f4aa01a8700 1 -- 172.21.15.178:0/4282793292 shutdown_connections 2021-11-20T06:00:00.292 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.303+0000 7f4aa01a8700 1 --2- 172.21.15.178:0/4282793292 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a98060220 0x7f4a980f9150 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:00.293 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.303+0000 7f4aa01a8700 1 -- 172.21.15.178:0/4282793292 >> 172.21.15.178:0/4282793292 conn(0x7f4a980f0b70 msgr2=0x7f4a980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:00.293 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.303+0000 7f4aa01a8700 1 -- 172.21.15.178:0/4282793292 shutdown_connections 2021-11-20T06:00:00.293 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.303+0000 7f4aa01a8700 1 -- 172.21.15.178:0/4282793292 wait complete. 2021-11-20T06:00:00.293 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.304+0000 7f4aa01a8700 1 Processor -- start 2021-11-20T06:00:00.293 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.304+0000 7f4aa01a8700 1 -- start start 2021-11-20T06:00:00.294 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.304+0000 7f4aa01a8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a98060220 0x7f4a980f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:00.294 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.304+0000 7f4aa01a8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980f5050 con 0x7f4a98060220 2021-11-20T06:00:00.294 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.304+0000 7f4a9df44700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a98060220 0x7f4a980f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:00.294 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.304+0000 7f4a9df44700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a98060220 0x7f4a980f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.178:33436/0 (socket says 172.21.15.178:33436) 2021-11-20T06:00:00.295 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.304+0000 7f4a9df44700 1 -- 172.21.15.178:0/2162837666 learned_addr learned my addr 172.21.15.178:0/2162837666 (peer_addr_for_me v2:172.21.15.178:0/0) 2021-11-20T06:00:00.295 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.304+0000 7f4a9df44700 1 -- 172.21.15.178:0/2162837666 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a88009640 con 0x7f4a98060220 2021-11-20T06:00:00.295 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.305+0000 7f4a9df44700 1 --2- 172.21.15.178:0/2162837666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a98060220 0x7f4a980f6bd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f4a88007bd0 tx=0x7f4a88004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:00.296 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.305+0000 7f4a967fc700 1 -- 172.21.15.178:0/2162837666 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a88018070 con 0x7f4a98060220 2021-11-20T06:00:00.296 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.305+0000 7f4a967fc700 1 -- 172.21.15.178:0/2162837666 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a8800b870 con 0x7f4a98060220 2021-11-20T06:00:00.296 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.305+0000 7f4aa01a8700 1 -- 172.21.15.178:0/2162837666 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a980f51f0 con 0x7f4a98060220 2021-11-20T06:00:00.296 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.305+0000 7f4a967fc700 1 -- 172.21.15.178:0/2162837666 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a8801d420 con 0x7f4a98060220 2021-11-20T06:00:00.296 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.305+0000 7f4aa01a8700 1 -- 172.21.15.178:0/2162837666 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a980f5680 con 0x7f4a98060220 2021-11-20T06:00:00.299 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.305+0000 7f4a967fc700 1 -- 172.21.15.178:0/2162837666 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a88024060 con 0x7f4a98060220 2021-11-20T06:00:00.299 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.305+0000 7f4aa01a8700 1 -- 172.21.15.178:0/2162837666 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a98059100 con 0x7f4a98060220 2021-11-20T06:00:00.299 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.306+0000 7f4a967fc700 1 --2- 172.21.15.178:0/2162837666 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4a8002ff60 0x7f4a80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:00.299 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.306+0000 7f4a967fc700 1 -- 172.21.15.178:0/2162837666 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a88041250 con 0x7f4a98060220 2021-11-20T06:00:00.300 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.306+0000 7f4a9d743700 1 --2- 172.21.15.178:0/2162837666 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4a8002ff60 0x7f4a80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:00.300 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.306+0000 7f4a9d743700 1 --2- 172.21.15.178:0/2162837666 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4a8002ff60 0x7f4a80032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f4a8c0096a0 tx=0x7f4a8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:00.300 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.312+0000 7f4a967fc700 1 -- 172.21.15.178:0/2162837666 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a88011790 con 0x7f4a98060220 2021-11-20T06:00:00.503 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.516+0000 7f4aa01a8700 1 -- 172.21.15.178:0/2162837666 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a98045ad0 con 0x7f4a98060220 2021-11-20T06:00:00.503 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.517+0000 7f4a967fc700 1 -- 172.21.15.178:0/2162837666 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4a88011790 con 0x7f4a98060220 2021-11-20T06:00:00.504 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T06:00:00.504 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"d819f2cc-49c6-11ec-8c2c-001a4aab830c","modified":"2021-11-20T05:58:31.415306Z","created":"2021-11-20T05:58:31.415306Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi139","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T06:00:00.505 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.518+0000 7f4aa01a8700 1 -- 172.21.15.178:0/2162837666 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4a8002ff60 msgr2=0x7f4a80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:00.505 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.518+0000 7f4aa01a8700 1 --2- 172.21.15.178:0/2162837666 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4a8002ff60 0x7f4a80032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f4a8c0096a0 tx=0x7f4a8c006b40).stop 2021-11-20T06:00:00.506 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.518+0000 7f4aa01a8700 1 -- 172.21.15.178:0/2162837666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a98060220 msgr2=0x7f4a980f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:00.506 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.518+0000 7f4aa01a8700 1 --2- 172.21.15.178:0/2162837666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a98060220 0x7f4a980f6bd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f4a88007bd0 tx=0x7f4a88004d10).stop 2021-11-20T06:00:00.506 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.518+0000 7f4aa01a8700 1 -- 172.21.15.178:0/2162837666 shutdown_connections 2021-11-20T06:00:00.506 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.518+0000 7f4aa01a8700 1 --2- 172.21.15.178:0/2162837666 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4a8002ff60 0x7f4a80032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:00.507 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.518+0000 7f4aa01a8700 1 --2- 172.21.15.178:0/2162837666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4a98060220 0x7f4a980f6bd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:00.507 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.518+0000 7f4aa01a8700 1 -- 172.21.15.178:0/2162837666 >> 172.21.15.178:0/2162837666 conn(0x7f4a980f0b70 msgr2=0x7f4a980f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:00.507 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.518+0000 7f4aa01a8700 1 -- 172.21.15.178:0/2162837666 shutdown_connections 2021-11-20T06:00:00.507 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:00.518+0000 7f4aa01a8700 1 -- 172.21.15.178:0/2162837666 wait complete. 2021-11-20T06:00:00.507 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2021-11-20T06:00:01.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:01 smithi139 conmon[33501]: cluster 2021-11-20T06:00:00.174588+0000 mgr.smithi139.dplyxn (mgr.14162) 23 2021-11-20T06:00:01.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:01 smithi139 conmon[33501]: : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:01.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:01 smithi139 conmon[33501]: audit 2021-11-20T06:00:00.517798+0000 mon.smithi139 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.178:0/2162837666' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T06:00:01.961 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T06:00:01.961 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T06:00:03.565 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.576+0000 7f25e995a700 1 -- 172.21.15.178:0/1208349601 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25e40f5410 msgr2=0x7f25e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:03.565 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.576+0000 7f25e995a700 1 --2- 172.21.15.178:0/1208349601 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25e40f5410 0x7f25e40f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f25d4009fb0 tx=0x7f25d4009b30).stop 2021-11-20T06:00:03.565 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.576+0000 7f25e995a700 1 -- 172.21.15.178:0/1208349601 shutdown_connections 2021-11-20T06:00:03.566 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.576+0000 7f25e995a700 1 --2- 172.21.15.178:0/1208349601 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25e40f5410 0x7f25e40f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:03.566 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.576+0000 7f25e995a700 1 -- 172.21.15.178:0/1208349601 >> 172.21.15.178:0/1208349601 conn(0x7f25e40f0b70 msgr2=0x7f25e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:03.566 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.577+0000 7f25e995a700 1 -- 172.21.15.178:0/1208349601 shutdown_connections 2021-11-20T06:00:03.566 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.577+0000 7f25e995a700 1 -- 172.21.15.178:0/1208349601 wait complete. 2021-11-20T06:00:03.567 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.577+0000 7f25e995a700 1 Processor -- start 2021-11-20T06:00:03.567 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.577+0000 7f25e995a700 1 -- start start 2021-11-20T06:00:03.567 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.577+0000 7f25e995a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25e40f5410 0x7f25e41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:03.567 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.577+0000 7f25e995a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e4106bd0 con 0x7f25e40f5410 2021-11-20T06:00:03.568 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.578+0000 7f25e2ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25e40f5410 0x7f25e41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:03.568 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.578+0000 7f25e2ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25e40f5410 0x7f25e41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.178:33442/0 (socket says 172.21.15.178:33442) 2021-11-20T06:00:03.568 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.578+0000 7f25e2ffd700 1 -- 172.21.15.178:0/2215755643 learned_addr learned my addr 172.21.15.178:0/2215755643 (peer_addr_for_me v2:172.21.15.178:0/0) 2021-11-20T06:00:03.568 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.578+0000 7f25e2ffd700 1 -- 172.21.15.178:0/2215755643 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d4005040 con 0x7f25e40f5410 2021-11-20T06:00:03.569 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.578+0000 7f25e2ffd700 1 --2- 172.21.15.178:0/2215755643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25e40f5410 0x7f25e41066e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f25d4013510 tx=0x7f25d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:03.569 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.578+0000 7f25cb7fe700 1 -- 172.21.15.178:0/2215755643 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25d4009320 con 0x7f25e40f5410 2021-11-20T06:00:03.569 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.578+0000 7f25cb7fe700 1 -- 172.21.15.178:0/2215755643 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25d4013f00 con 0x7f25e40f5410 2021-11-20T06:00:03.569 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.578+0000 7f25e995a700 1 -- 172.21.15.178:0/2215755643 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e4106dd0 con 0x7f25e40f5410 2021-11-20T06:00:03.570 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.578+0000 7f25cb7fe700 1 -- 172.21.15.178:0/2215755643 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25d4014d90 con 0x7f25e40f5410 2021-11-20T06:00:03.570 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.578+0000 7f25e995a700 1 -- 172.21.15.178:0/2215755643 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e4107260 con 0x7f25e40f5410 2021-11-20T06:00:03.571 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.579+0000 7f25cb7fe700 1 -- 172.21.15.178:0/2215755643 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25d4014420 con 0x7f25e40f5410 2021-11-20T06:00:03.571 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.579+0000 7f25e995a700 1 -- 172.21.15.178:0/2215755643 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e4045ad0 con 0x7f25e40f5410 2021-11-20T06:00:03.572 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.579+0000 7f25cb7fe700 1 --2- 172.21.15.178:0/2215755643 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f25cc030000 0x7f25cc0324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:03.573 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.579+0000 7f25cb7fe700 1 -- 172.21.15.178:0/2215755643 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25d4041090 con 0x7f25e40f5410 2021-11-20T06:00:03.573 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.579+0000 7f25e27fc700 1 --2- 172.21.15.178:0/2215755643 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f25cc030000 0x7f25cc0324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:03.573 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.580+0000 7f25e27fc700 1 --2- 172.21.15.178:0/2215755643 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f25cc030000 0x7f25cc0324c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f25d80096a0 tx=0x7f25d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:03.573 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.584+0000 7f25cb7fe700 1 -- 172.21.15.178:0/2215755643 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25d406d2b0 con 0x7f25e40f5410 2021-11-20T06:00:03.773 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.786+0000 7f25e995a700 1 -- 172.21.15.178:0/2215755643 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25e40fae60 con 0x7f25e40f5410 2021-11-20T06:00:03.773 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.787+0000 7f25cb7fe700 1 -- 172.21.15.178:0/2215755643 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f25d40095f0 con 0x7f25e40f5410 2021-11-20T06:00:03.774 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T06:00:03.774 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"d819f2cc-49c6-11ec-8c2c-001a4aab830c","modified":"2021-11-20T05:58:31.415306Z","created":"2021-11-20T05:58:31.415306Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi139","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T06:00:03.775 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.789+0000 7f25e995a700 1 -- 172.21.15.178:0/2215755643 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f25cc030000 msgr2=0x7f25cc0324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:03.776 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.789+0000 7f25e995a700 1 --2- 172.21.15.178:0/2215755643 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f25cc030000 0x7f25cc0324c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f25d80096a0 tx=0x7f25d8006b40).stop 2021-11-20T06:00:03.776 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.789+0000 7f25e995a700 1 -- 172.21.15.178:0/2215755643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25e40f5410 msgr2=0x7f25e41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:03.776 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.789+0000 7f25e995a700 1 --2- 172.21.15.178:0/2215755643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25e40f5410 0x7f25e41066e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f25d4013510 tx=0x7f25d400b040).stop 2021-11-20T06:00:03.777 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.789+0000 7f25e995a700 1 -- 172.21.15.178:0/2215755643 shutdown_connections 2021-11-20T06:00:03.777 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.789+0000 7f25e995a700 1 --2- 172.21.15.178:0/2215755643 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f25cc030000 0x7f25cc0324c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:03.777 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.789+0000 7f25e995a700 1 --2- 172.21.15.178:0/2215755643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25e40f5410 0x7f25e41066e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:03.777 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.789+0000 7f25e995a700 1 -- 172.21.15.178:0/2215755643 >> 172.21.15.178:0/2215755643 conn(0x7f25e40f0b70 msgr2=0x7f25e405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:03.777 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.789+0000 7f25e995a700 1 -- 172.21.15.178:0/2215755643 shutdown_connections 2021-11-20T06:00:03.778 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:03.789+0000 7f25e995a700 1 -- 172.21.15.178:0/2215755643 wait complete. 2021-11-20T06:00:03.778 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2021-11-20T06:00:04.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:04 smithi139 conmon[33501]: cluster 2021-11-20T06:00:02.174795+0000 mgr.smithi139.dplyxn (mgr.14162) 24 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:04.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:04 smithi139 conmon[33501]: audit 2021-11-20T06:00:03.787906+0000 mon.smithi139 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.178:0/2215755643' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T06:00:05.209 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T06:00:05.210 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T06:00:06.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:06 smithi139 conmon[33501]: cluster 2021-11-20T06:00:04.175048+0000 mgr.smithi139.dplyxn (mgr.14162) 25 2021-11-20T06:00:06.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:06 smithi139 conmon[33501]: : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:07.162 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.173+0000 7fdd75460700 1 -- 172.21.15.178:0/2217955104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd700f62d0 msgr2=0x7fdd700f66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:07.163 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.173+0000 7fdd75460700 1 --2- 172.21.15.178:0/2217955104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd700f62d0 0x7fdd700f66f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fdd60009fb0 tx=0x7fdd60009b30).stop 2021-11-20T06:00:07.163 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.173+0000 7fdd75460700 1 -- 172.21.15.178:0/2217955104 shutdown_connections 2021-11-20T06:00:07.163 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.173+0000 7fdd75460700 1 --2- 172.21.15.178:0/2217955104 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd700f62d0 0x7fdd700f66f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:07.163 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.173+0000 7fdd75460700 1 -- 172.21.15.178:0/2217955104 >> 172.21.15.178:0/2217955104 conn(0x7fdd700f1a30 msgr2=0x7fdd700f3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:07.164 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.173+0000 7fdd75460700 1 -- 172.21.15.178:0/2217955104 shutdown_connections 2021-11-20T06:00:07.164 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.173+0000 7fdd75460700 1 -- 172.21.15.178:0/2217955104 wait complete. 2021-11-20T06:00:07.164 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.174+0000 7fdd75460700 1 Processor -- start 2021-11-20T06:00:07.164 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.174+0000 7fdd75460700 1 -- start start 2021-11-20T06:00:07.165 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.174+0000 7fdd75460700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd700f62d0 0x7fdd70106560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:07.165 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.174+0000 7fdd75460700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd70106a50 con 0x7fdd700f62d0 2021-11-20T06:00:07.165 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.174+0000 7fdd6effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd700f62d0 0x7fdd70106560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:07.165 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.175+0000 7fdd6effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd700f62d0 0x7fdd70106560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.178:33448/0 (socket says 172.21.15.178:33448) 2021-11-20T06:00:07.166 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.175+0000 7fdd6effd700 1 -- 172.21.15.178:0/2992655738 learned_addr learned my addr 172.21.15.178:0/2992655738 (peer_addr_for_me v2:172.21.15.178:0/0) 2021-11-20T06:00:07.166 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.175+0000 7fdd6effd700 1 -- 172.21.15.178:0/2992655738 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd60005040 con 0x7fdd700f62d0 2021-11-20T06:00:07.166 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.175+0000 7fdd6effd700 1 --2- 172.21.15.178:0/2992655738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd700f62d0 0x7fdd70106560 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fdd60014000 tx=0x7fdd6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:07.166 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.175+0000 7fdd577fe700 1 -- 172.21.15.178:0/2992655738 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd600095f0 con 0x7fdd700f62d0 2021-11-20T06:00:07.166 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.175+0000 7fdd577fe700 1 -- 172.21.15.178:0/2992655738 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd60007e70 con 0x7fdd700f62d0 2021-11-20T06:00:07.167 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.175+0000 7fdd577fe700 1 -- 172.21.15.178:0/2992655738 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd60014ea0 con 0x7fdd700f62d0 2021-11-20T06:00:07.167 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.175+0000 7fdd75460700 1 -- 172.21.15.178:0/2992655738 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd70106c50 con 0x7fdd700f62d0 2021-11-20T06:00:07.167 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.175+0000 7fdd75460700 1 -- 172.21.15.178:0/2992655738 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd701070e0 con 0x7fdd700f62d0 2021-11-20T06:00:07.167 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.176+0000 7fdd577fe700 1 -- 172.21.15.178:0/2992655738 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd60025070 con 0x7fdd700f62d0 2021-11-20T06:00:07.168 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.176+0000 7fdd577fe700 1 --2- 172.21.15.178:0/2992655738 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdd5802ff60 0x7fdd58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:07.168 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.176+0000 7fdd577fe700 1 -- 172.21.15.178:0/2992655738 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd60040a70 con 0x7fdd700f62d0 2021-11-20T06:00:07.168 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.176+0000 7fdd75460700 1 -- 172.21.15.178:0/2992655738 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd70045ad0 con 0x7fdd700f62d0 2021-11-20T06:00:07.168 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.176+0000 7fdd6e7fc700 1 --2- 172.21.15.178:0/2992655738 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdd5802ff60 0x7fdd58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:07.169 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.177+0000 7fdd6e7fc700 1 --2- 172.21.15.178:0/2992655738 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdd5802ff60 0x7fdd58032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fdd640096a0 tx=0x7fdd64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:07.169 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.181+0000 7fdd577fe700 1 -- 172.21.15.178:0/2992655738 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd60026350 con 0x7fdd700f62d0 2021-11-20T06:00:07.383 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.397+0000 7fdd75460700 1 -- 172.21.15.178:0/2992655738 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd700fbe80 con 0x7fdd700f62d0 2021-11-20T06:00:07.384 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.398+0000 7fdd577fe700 1 -- 172.21.15.178:0/2992655738 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdd60012330 con 0x7fdd700f62d0 2021-11-20T06:00:07.384 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T06:00:07.385 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"d819f2cc-49c6-11ec-8c2c-001a4aab830c","modified":"2021-11-20T05:58:31.415306Z","created":"2021-11-20T05:58:31.415306Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi139","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T06:00:07.386 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.400+0000 7fdd75460700 1 -- 172.21.15.178:0/2992655738 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdd5802ff60 msgr2=0x7fdd58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:07.386 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.400+0000 7fdd75460700 1 --2- 172.21.15.178:0/2992655738 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdd5802ff60 0x7fdd58032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fdd640096a0 tx=0x7fdd64006b40).stop 2021-11-20T06:00:07.387 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.400+0000 7fdd75460700 1 -- 172.21.15.178:0/2992655738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd700f62d0 msgr2=0x7fdd70106560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:07.387 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.400+0000 7fdd75460700 1 --2- 172.21.15.178:0/2992655738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd700f62d0 0x7fdd70106560 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fdd60014000 tx=0x7fdd6000b040).stop 2021-11-20T06:00:07.387 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.400+0000 7fdd75460700 1 -- 172.21.15.178:0/2992655738 shutdown_connections 2021-11-20T06:00:07.387 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.400+0000 7fdd75460700 1 --2- 172.21.15.178:0/2992655738 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdd5802ff60 0x7fdd58032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:07.388 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.400+0000 7fdd75460700 1 --2- 172.21.15.178:0/2992655738 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdd700f62d0 0x7fdd70106560 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:07.388 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.400+0000 7fdd75460700 1 -- 172.21.15.178:0/2992655738 >> 172.21.15.178:0/2992655738 conn(0x7fdd700f1a30 msgr2=0x7fdd700f2520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:07.388 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.400+0000 7fdd75460700 1 -- 172.21.15.178:0/2992655738 shutdown_connections 2021-11-20T06:00:07.388 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:07.400+0000 7fdd75460700 1 -- 172.21.15.178:0/2992655738 wait complete. 2021-11-20T06:00:07.389 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2021-11-20T06:00:07.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:07 smithi139 conmon[33501]: audit 2021-11-20T06:00:06.140749+0000 mon.smithi139 ( 2021-11-20T06:00:07.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:07 smithi139 conmon[33501]: mon.0) 134 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:07.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:07 smithi139 conmon[33501]: audit 2021-11-20T06:00:06.141267+0000 mon.smithi139 (mon.0) 135 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:00:07.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:07 smithi139 conmon[33501]: audit 2021-11-20T06:00:06.531788+0000 mon.smithi139 (mon.0) 136 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:08.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:08 smithi139 conmon[33501]: cluster 2021-11-20T06:00:06.175274+0000 mgr.smithi139.dplyxn (mgr.14162) 26 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:08.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:08 smithi139 conmon[33501]: audit 2021-11-20T06:00:07.168678+0000 mon.smithi139 (mon.0) 137 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:08.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:08 smithi139 conmon[33501]: audit 2021-11-20T06:00:07.398459+0000 mon.smithi139 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.178:0/2992655738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T06:00:09.056 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T06:00:09.057 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T06:00:10.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:10 smithi139 conmon[33501]: cluster 2021-11-20T06:00:08.175492+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:00:10.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:10 smithi139 conmon[33501]: 14162) 27 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:11.109 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.122+0000 7f3a2cde6700 1 -- 172.21.15.178:0/2385638624 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a280f5410 msgr2=0x7f3a280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:11.109 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.122+0000 7f3a2cde6700 1 --2- 172.21.15.178:0/2385638624 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a280f5410 0x7f3a280f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f3a1c009fb0 tx=0x7f3a1c009b30).stop 2021-11-20T06:00:11.109 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.122+0000 7f3a2cde6700 1 -- 172.21.15.178:0/2385638624 shutdown_connections 2021-11-20T06:00:11.110 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.122+0000 7f3a2cde6700 1 --2- 172.21.15.178:0/2385638624 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a280f5410 0x7f3a280f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:11.110 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.122+0000 7f3a2cde6700 1 -- 172.21.15.178:0/2385638624 >> 172.21.15.178:0/2385638624 conn(0x7f3a280f0b70 msgr2=0x7f3a280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:11.110 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.122+0000 7f3a2cde6700 1 -- 172.21.15.178:0/2385638624 shutdown_connections 2021-11-20T06:00:11.111 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.122+0000 7f3a2cde6700 1 -- 172.21.15.178:0/2385638624 wait complete. 2021-11-20T06:00:11.112 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.123+0000 7f3a2cde6700 1 Processor -- start 2021-11-20T06:00:11.112 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.123+0000 7f3a2cde6700 1 -- start start 2021-11-20T06:00:11.115 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.123+0000 7f3a2cde6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a280f5410 0x7f3a281055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:11.115 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.123+0000 7f3a2cde6700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a28105ac0 con 0x7f3a280f5410 2021-11-20T06:00:11.116 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.123+0000 7f3a2659c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a280f5410 0x7f3a281055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:11.116 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.123+0000 7f3a2659c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a280f5410 0x7f3a281055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.178:33454/0 (socket says 172.21.15.178:33454) 2021-11-20T06:00:11.116 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.123+0000 7f3a2659c700 1 -- 172.21.15.178:0/1705866301 learned_addr learned my addr 172.21.15.178:0/1705866301 (peer_addr_for_me v2:172.21.15.178:0/0) 2021-11-20T06:00:11.116 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.124+0000 7f3a2659c700 1 -- 172.21.15.178:0/1705866301 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c005040 con 0x7f3a280f5410 2021-11-20T06:00:11.117 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.124+0000 7f3a2659c700 1 --2- 172.21.15.178:0/1705866301 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a280f5410 0x7f3a281055d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3a1c013510 tx=0x7f3a1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:11.117 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.124+0000 7f3a16ffd700 1 -- 172.21.15.178:0/1705866301 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a1c009320 con 0x7f3a280f5410 2021-11-20T06:00:11.117 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.124+0000 7f3a16ffd700 1 -- 172.21.15.178:0/1705866301 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a1c013f00 con 0x7f3a280f5410 2021-11-20T06:00:11.117 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.124+0000 7f3a16ffd700 1 -- 172.21.15.178:0/1705866301 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a1c014d90 con 0x7f3a280f5410 2021-11-20T06:00:11.117 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.124+0000 7f3a2cde6700 1 -- 172.21.15.178:0/1705866301 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a28105cc0 con 0x7f3a280f5410 2021-11-20T06:00:11.118 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.124+0000 7f3a2cde6700 1 -- 172.21.15.178:0/1705866301 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a28106150 con 0x7f3a280f5410 2021-11-20T06:00:11.121 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.125+0000 7f3a16ffd700 1 -- 172.21.15.178:0/1705866301 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a1c0263a0 con 0x7f3a280f5410 2021-11-20T06:00:11.123 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.125+0000 7f3a16ffd700 1 --2- 172.21.15.178:0/1705866301 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3a1002ff60 0x7f3a10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:11.123 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.125+0000 7f3a16ffd700 1 -- 172.21.15.178:0/1705866301 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a1c040ff0 con 0x7f3a280f5410 2021-11-20T06:00:11.124 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.125+0000 7f3a2cde6700 1 -- 172.21.15.178:0/1705866301 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a28045ad0 con 0x7f3a280f5410 2021-11-20T06:00:11.124 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.125+0000 7f3a25d9b700 1 --2- 172.21.15.178:0/1705866301 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3a1002ff60 0x7f3a10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:11.124 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.126+0000 7f3a25d9b700 1 --2- 172.21.15.178:0/1705866301 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3a1002ff60 0x7f3a10032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f3a180096a0 tx=0x7f3a18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:11.125 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.130+0000 7f3a16ffd700 1 -- 172.21.15.178:0/1705866301 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a1c012920 con 0x7f3a280f5410 2021-11-20T06:00:11.344 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.358+0000 7f3a2cde6700 1 -- 172.21.15.178:0/1705866301 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a28106e70 con 0x7f3a280f5410 2021-11-20T06:00:11.347 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.359+0000 7f3a16ffd700 1 -- 172.21.15.178:0/1705866301 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3a1c02b020 con 0x7f3a280f5410 2021-11-20T06:00:11.347 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T06:00:11.348 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"d819f2cc-49c6-11ec-8c2c-001a4aab830c","modified":"2021-11-20T05:58:31.415306Z","created":"2021-11-20T05:58:31.415306Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi139","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T06:00:11.349 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.361+0000 7f3a2cde6700 1 -- 172.21.15.178:0/1705866301 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3a1002ff60 msgr2=0x7f3a10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:11.350 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.361+0000 7f3a2cde6700 1 --2- 172.21.15.178:0/1705866301 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3a1002ff60 0x7f3a10032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f3a180096a0 tx=0x7f3a18006b40).stop 2021-11-20T06:00:11.350 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.361+0000 7f3a2cde6700 1 -- 172.21.15.178:0/1705866301 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a280f5410 msgr2=0x7f3a281055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:11.351 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.361+0000 7f3a2cde6700 1 --2- 172.21.15.178:0/1705866301 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a280f5410 0x7f3a281055d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3a1c013510 tx=0x7f3a1c00b040).stop 2021-11-20T06:00:11.351 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.361+0000 7f3a2cde6700 1 -- 172.21.15.178:0/1705866301 shutdown_connections 2021-11-20T06:00:11.351 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.361+0000 7f3a2cde6700 1 --2- 172.21.15.178:0/1705866301 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3a1002ff60 0x7f3a10032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:11.351 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.361+0000 7f3a2cde6700 1 --2- 172.21.15.178:0/1705866301 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3a280f5410 0x7f3a281055d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:11.352 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.361+0000 7f3a2cde6700 1 -- 172.21.15.178:0/1705866301 >> 172.21.15.178:0/1705866301 conn(0x7f3a280f0b70 msgr2=0x7f3a280f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:11.352 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.361+0000 7f3a2cde6700 1 -- 172.21.15.178:0/1705866301 shutdown_connections 2021-11-20T06:00:11.352 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:11.361+0000 7f3a2cde6700 1 -- 172.21.15.178:0/1705866301 wait complete. 2021-11-20T06:00:11.352 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2021-11-20T06:00:11.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:11 smithi139 conmon[33501]: cluster 2021-11-20T06:00:10.175784+0000 mgr.smithi139.dplyxn (mgr.14162) 28 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:11.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:11 smithi139 conmon[33501]: audit 2021-11-20T06:00:11.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:11 smithi139 conmon[33501]: 2021-11-20T06:00:10.312575+0000 mon.smithi139 (mon.0) 139 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:11.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:11 smithi139 conmon[33501]: audit 2021-11-20T06:00:10.556312+0000 mon.smithi139 (mon.0) 140 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:11.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:11 smithi139 conmon[33501]: audit 2021-11-20T06:00:10.976196+0000 mon.smithi139 (mon.0) 141 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:11.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:11 smithi139 conmon[33501]: audit 2021-11-20T06:00:10.979641+0000 mon.smithi139 (mon 2021-11-20T06:00:11.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:11 smithi139 conmon[33501]: .0) 142 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:11.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:11 smithi139 conmon[33501]: audit 2021-11-20T06:00:10.980875+0000 mon.smithi139 (mon.0) 143 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi178", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T06:00:11.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:11 smithi139 conmon[33501]: audit 2021-11-20T06:00:11.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:11 smithi139 conmon[33501]: 2021-11-20T06:00:10.983553+0000 mon.smithi139 (mon.0) 144 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi178", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2021-11-20T06:00:11.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:11 smithi139 conmon[33501]: audit 2021-11-20T 2021-11-20T06:00:11.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:11 smithi139 conmon[33501]: 06:00:10.984378+0000 mon.smithi139 (mon.0) 145 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:00:11.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:11 smithi139 conmon[33501]: cephadm 2021-11-20T06:00:11.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:11 smithi139 conmon[33501]: 2021-11-20T06:00:10.985084+0000 mgr.smithi139.dplyxn (mgr.14162) 29 : cephadm [INF] Deploying daemon crash.smithi178 on smithi178 2021-11-20T06:00:12.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:12 smithi139 conmon[33501]: audit 2021-11-20T06:00:11.359808+0000 mon.smithi139 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.178:0/1705866301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T06:00:12.804 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T06:00:12.804 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T06:00:14.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:14 smithi139 conmon[33501]: cluster 2021-11-20T06:00:12.176035+0000 mgr.smithi139.dplyxn (mgr.14162) 30 2021-11-20T06:00:14.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:14 smithi139 conmon[33501]: : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:16.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:16 smithi139 conmon[33501]: cluster 2021-11-20T06:00:14.176441+0000 mgr.smithi139.dplyxn (mgr.14162) 31 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:18.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:18 smithi139 conmon[33501]: cluster 2021-11-20T06:00:16.176799+0000 mgr.smithi139.dplyxn (mgr.14162) 32 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:18.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:18 smithi139 conmon[33501]: 2021-11-20T06:00:18.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:18 smithi139 conmon[33501]: audit 2021-11-20T06:00:17.148594+0000 mon.smithi139 (mon.0) 147 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:18.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:18 smithi139 conmon[33501]: audit 2021-11-20T06:00:17.149920+0000 mon.smithi139 (mon.0) 148 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi178.zxxdkz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T06:00:18.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:18 smithi139 conmon[33501]: audit 2021-11-20T06:00:18.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:18 smithi139 conmon[33501]: 2021-11-20T06:00:17.154576+0000 mon.smithi139 (mon.0) 149 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi178.zxxdkz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-11-20T06:00:18.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:18 smithi139 conmon[33501]: audit 2021-11-20T06:00:17.155365+0000 mon.smithi139 (mon. 2021-11-20T06:00:18.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:18 smithi139 conmon[33501]: 0) 150 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T06:00:18.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:18 smithi139 conmon[33501]: audit 2021-11-20T06: 2021-11-20T06:00:18.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:18 smithi139 conmon[33501]: 00:17.156338+0000 mon.smithi139 (mon.0) 151 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:00:19.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:19 smithi139 conmon[33501]: cephadm 2021-11-20T06:00:17.156994+0000 mgr.smithi139.dplyxn (mgr.14162) 33 : cephadm [INF] Deploying daemon mgr.smithi178.zxxdkz on smithi178 2021-11-20T06:00:20.485 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.376+0000 7fe7fec3f700 1 -- 172.21.15.178:0/1547444965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7f80f6510 msgr2=0x7fe7f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:20.485 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.376+0000 7fe7fec3f700 1 --2- 172.21.15.178:0/1547444965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7f80f6510 0x7fe7f80f6930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fe7ec009fb0 tx=0x7fe7ec009b30).stop 2021-11-20T06:00:20.486 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.376+0000 7fe7fec3f700 1 -- 172.21.15.178:0/1547444965 shutdown_connections 2021-11-20T06:00:20.486 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.376+0000 7fe7fec3f700 1 --2- 172.21.15.178:0/1547444965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7f80f6510 0x7fe7f80f6930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:20.486 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.376+0000 7fe7fec3f700 1 -- 172.21.15.178:0/1547444965 >> 172.21.15.178:0/1547444965 conn(0x7fe7f80f1c70 msgr2=0x7fe7f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:20.486 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.376+0000 7fe7fec3f700 1 -- 172.21.15.178:0/1547444965 shutdown_connections 2021-11-20T06:00:20.487 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.376+0000 7fe7fec3f700 1 -- 172.21.15.178:0/1547444965 wait complete. 2021-11-20T06:00:20.487 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.377+0000 7fe7fec3f700 1 Processor -- start 2021-11-20T06:00:20.487 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.377+0000 7fe7fec3f700 1 -- start start 2021-11-20T06:00:20.487 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.377+0000 7fe7fec3f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7f80f6510 0x7fe7f806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:20.487 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.377+0000 7fe7fec3f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f8069e20 con 0x7fe7f80f6510 2021-11-20T06:00:20.488 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.378+0000 7fe7fc9db700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7f80f6510 0x7fe7f806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:20.489 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.378+0000 7fe7fc9db700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7f80f6510 0x7fe7f806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.178:33460/0 (socket says 172.21.15.178:33460) 2021-11-20T06:00:20.490 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.378+0000 7fe7fc9db700 1 -- 172.21.15.178:0/2115027486 learned_addr learned my addr 172.21.15.178:0/2115027486 (peer_addr_for_me v2:172.21.15.178:0/0) 2021-11-20T06:00:20.490 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.378+0000 7fe7fc9db700 1 -- 172.21.15.178:0/2115027486 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ec005040 con 0x7fe7f80f6510 2021-11-20T06:00:20.490 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.378+0000 7fe7fc9db700 1 --2- 172.21.15.178:0/2115027486 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7f80f6510 0x7fe7f806b9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fe7ec013510 tx=0x7fe7ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:20.491 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.378+0000 7fe7f57fa700 1 -- 172.21.15.178:0/2115027486 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7ec009320 con 0x7fe7f80f6510 2021-11-20T06:00:20.491 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.378+0000 7fe7f57fa700 1 -- 172.21.15.178:0/2115027486 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe7ec013f00 con 0x7fe7f80f6510 2021-11-20T06:00:20.491 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.378+0000 7fe7fec3f700 1 -- 172.21.15.178:0/2115027486 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f8069fc0 con 0x7fe7f80f6510 2021-11-20T06:00:20.491 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.378+0000 7fe7f57fa700 1 -- 172.21.15.178:0/2115027486 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7ec014d90 con 0x7fe7f80f6510 2021-11-20T06:00:20.492 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.378+0000 7fe7fec3f700 1 -- 172.21.15.178:0/2115027486 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f806a450 con 0x7fe7f80f6510 2021-11-20T06:00:20.492 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.379+0000 7fe7f57fa700 1 -- 172.21.15.178:0/2115027486 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7ec014420 con 0x7fe7f80f6510 2021-11-20T06:00:20.492 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.379+0000 7fe7fec3f700 1 -- 172.21.15.178:0/2115027486 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f8059100 con 0x7fe7f80f6510 2021-11-20T06:00:20.493 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.379+0000 7fe7f57fa700 1 --2- 172.21.15.178:0/2115027486 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe7e802ff60 0x7fe7e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:20.493 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.379+0000 7fe7f57fa700 1 -- 172.21.15.178:0/2115027486 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7ec041070 con 0x7fe7f80f6510 2021-11-20T06:00:20.493 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.380+0000 7fe7f7fff700 1 --2- 172.21.15.178:0/2115027486 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe7e802ff60 0x7fe7e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:20.493 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.380+0000 7fe7f7fff700 1 --2- 172.21.15.178:0/2115027486 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe7e802ff60 0x7fe7e8032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fe7e40096a0 tx=0x7fe7e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:20.494 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.384+0000 7fe7f57fa700 1 -- 172.21.15.178:0/2115027486 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7ec0095f0 con 0x7fe7f80f6510 2021-11-20T06:00:20.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:20 smithi139 conmon[33501]: cluster 2021-11-20T06:00:18.177179+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:00:20.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:20 smithi139 conmon[33501]: mgr.14162) 34 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:20.578 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.592+0000 7fe7fec3f700 1 -- 172.21.15.178:0/2115027486 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7f80fbf60 con 0x7fe7f80f6510 2021-11-20T06:00:20.580 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.592+0000 7fe7f57fa700 1 -- 172.21.15.178:0/2115027486 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe7ec01db70 con 0x7fe7f80f6510 2021-11-20T06:00:20.581 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T06:00:20.581 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"d819f2cc-49c6-11ec-8c2c-001a4aab830c","modified":"2021-11-20T05:58:31.415306Z","created":"2021-11-20T05:58:31.415306Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi139","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T06:00:20.583 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.594+0000 7fe7fec3f700 1 -- 172.21.15.178:0/2115027486 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe7e802ff60 msgr2=0x7fe7e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:20.584 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.594+0000 7fe7fec3f700 1 --2- 172.21.15.178:0/2115027486 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe7e802ff60 0x7fe7e8032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fe7e40096a0 tx=0x7fe7e4006b40).stop 2021-11-20T06:00:20.584 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.594+0000 7fe7fec3f700 1 -- 172.21.15.178:0/2115027486 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7f80f6510 msgr2=0x7fe7f806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:20.584 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.594+0000 7fe7fec3f700 1 --2- 172.21.15.178:0/2115027486 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7f80f6510 0x7fe7f806b9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fe7ec013510 tx=0x7fe7ec00b040).stop 2021-11-20T06:00:20.585 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.594+0000 7fe7fec3f700 1 -- 172.21.15.178:0/2115027486 shutdown_connections 2021-11-20T06:00:20.585 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.594+0000 7fe7fec3f700 1 --2- 172.21.15.178:0/2115027486 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe7e802ff60 0x7fe7e8032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:20.585 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.594+0000 7fe7fec3f700 1 --2- 172.21.15.178:0/2115027486 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7f80f6510 0x7fe7f806b9a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:20.585 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.594+0000 7fe7fec3f700 1 -- 172.21.15.178:0/2115027486 >> 172.21.15.178:0/2115027486 conn(0x7fe7f80f1c70 msgr2=0x7fe7f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:20.586 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.594+0000 7fe7fec3f700 1 -- 172.21.15.178:0/2115027486 shutdown_connections 2021-11-20T06:00:20.586 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:20.594+0000 7fe7fec3f700 1 -- 172.21.15.178:0/2115027486 wait complete. 2021-11-20T06:00:20.586 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2021-11-20T06:00:21.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:21 smithi139 conmon[33501]: cluster 2021-11-20T06:00:20.177527+0000 mgr.smithi139.dplyxn (mgr.14162) 35 : cluster 2021-11-20T06:00:21.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:21 smithi139 conmon[33501]: [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:21.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:21 smithi139 conmon[33501]: audit 2021-11-20T06:00:20.593346+0000 mon.smithi139 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.178:0/2115027486' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T06:00:22.716 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T06:00:22.717 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T06:00:24.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:23 smithi139 conmon[33501]: cluster 2021-11-20T06:00:22.177884+0000 mgr.smithi139.dplyxn (mgr.14162) 36 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:24.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:23 smithi139 conmon[33501]: audit 2021-11-20T06:00:22.857231+0000 mon.smithi139 ( 2021-11-20T06:00:24.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:23 smithi139 conmon[33501]: mon.0) 153 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:24.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:23 smithi139 conmon[33501]: audit 2021-11-20T06:00:22.858482+0000 mon.smithi139 (mon.0) 154 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T06:00:24.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:23 smithi139 conmon[33501]: audit 2021-11-20T06:00:22.859320+0000 mon.smithi139 (mon.0) 155 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:00:24.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:23 smithi139 conmon[33501]: cephadm 2021-11-20T06:00:22.860058+0000 mgr.smithi139.dplyxn 2021-11-20T06:00:24.280 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:23 smithi139 conmon[33501]: (mgr.14162) 37 : cephadm [INF] Deploying daemon mon.smithi178 on smithi178 2021-11-20T06:00:24.487 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.493+0000 7f38311d1700 1 -- 172.21.15.178:0/452839720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f382c0f6510 msgr2=0x7f382c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:24.487 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.493+0000 7f38311d1700 1 --2- 172.21.15.178:0/452839720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f382c0f6510 0x7f382c0f6930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f3814009fb0 tx=0x7f3814009b30).stop 2021-11-20T06:00:24.488 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.494+0000 7f38311d1700 1 -- 172.21.15.178:0/452839720 shutdown_connections 2021-11-20T06:00:24.488 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.494+0000 7f38311d1700 1 --2- 172.21.15.178:0/452839720 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f382c0f6510 0x7f382c0f6930 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:24.488 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.494+0000 7f38311d1700 1 -- 172.21.15.178:0/452839720 >> 172.21.15.178:0/452839720 conn(0x7f382c0f1c70 msgr2=0x7f382c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:24.488 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.494+0000 7f38311d1700 1 -- 172.21.15.178:0/452839720 shutdown_connections 2021-11-20T06:00:24.488 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.494+0000 7f38311d1700 1 -- 172.21.15.178:0/452839720 wait complete. 2021-11-20T06:00:24.489 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.494+0000 7f38311d1700 1 Processor -- start 2021-11-20T06:00:24.489 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.494+0000 7f38311d1700 1 -- start start 2021-11-20T06:00:24.489 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.495+0000 7f38311d1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f382c0f6510 0x7f382c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:24.489 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.495+0000 7f38311d1700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c069e20 con 0x7f382c0f6510 2021-11-20T06:00:24.490 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.495+0000 7f382ad9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f382c0f6510 0x7f382c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:24.490 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.495+0000 7f382ad9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f382c0f6510 0x7f382c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.178:33470/0 (socket says 172.21.15.178:33470) 2021-11-20T06:00:24.490 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.495+0000 7f382ad9d700 1 -- 172.21.15.178:0/2623473597 learned_addr learned my addr 172.21.15.178:0/2623473597 (peer_addr_for_me v2:172.21.15.178:0/0) 2021-11-20T06:00:24.490 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.495+0000 7f382ad9d700 1 -- 172.21.15.178:0/2623473597 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3814005040 con 0x7f382c0f6510 2021-11-20T06:00:24.491 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.495+0000 7f382ad9d700 1 --2- 172.21.15.178:0/2623473597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f382c0f6510 0x7f382c06b9a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3814013510 tx=0x7f381400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:24.491 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.495+0000 7f38237fe700 1 -- 172.21.15.178:0/2623473597 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3814009320 con 0x7f382c0f6510 2021-11-20T06:00:24.491 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.495+0000 7f38311d1700 1 -- 172.21.15.178:0/2623473597 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382c069fc0 con 0x7f382c0f6510 2021-11-20T06:00:24.491 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.495+0000 7f38237fe700 1 -- 172.21.15.178:0/2623473597 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3814013f00 con 0x7f382c0f6510 2021-11-20T06:00:24.491 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.496+0000 7f38237fe700 1 -- 172.21.15.178:0/2623473597 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3814014d90 con 0x7f382c0f6510 2021-11-20T06:00:24.492 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.496+0000 7f38311d1700 1 -- 172.21.15.178:0/2623473597 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382c06a450 con 0x7f382c0f6510 2021-11-20T06:00:24.492 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.496+0000 7f38237fe700 1 -- 172.21.15.178:0/2623473597 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38140263a0 con 0x7f382c0f6510 2021-11-20T06:00:24.492 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.496+0000 7f38311d1700 1 -- 172.21.15.178:0/2623473597 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382c059100 con 0x7f382c0f6510 2021-11-20T06:00:24.493 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.496+0000 7f38237fe700 1 --2- 172.21.15.178:0/2623473597 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f381802ff60 0x7f3818032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:24.493 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.497+0000 7f38237fe700 1 -- 172.21.15.178:0/2623473597 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38140413c0 con 0x7f382c0f6510 2021-11-20T06:00:24.493 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.497+0000 7f382a59c700 1 --2- 172.21.15.178:0/2623473597 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f381802ff60 0x7f3818032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:24.494 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.497+0000 7f382a59c700 1 --2- 172.21.15.178:0/2623473597 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f381802ff60 0x7f3818032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f381c0096a0 tx=0x7f381c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:24.494 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.501+0000 7f38237fe700 1 -- 172.21.15.178:0/2623473597 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38140126f0 con 0x7f382c0f6510 2021-11-20T06:00:24.717 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.730+0000 7f38311d1700 1 -- 172.21.15.178:0/2623473597 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f382c0fbf60 con 0x7f382c0f6510 2021-11-20T06:00:24.718 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.731+0000 7f38237fe700 1 -- 172.21.15.178:0/2623473597 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3814012330 con 0x7f382c0f6510 2021-11-20T06:00:24.718 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T06:00:24.718 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"d819f2cc-49c6-11ec-8c2c-001a4aab830c","modified":"2021-11-20T05:58:31.415306Z","created":"2021-11-20T05:58:31.415306Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi139","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-20T06:00:24.719 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.733+0000 7f38311d1700 1 -- 172.21.15.178:0/2623473597 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f381802ff60 msgr2=0x7f3818032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:24.720 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.733+0000 7f38311d1700 1 --2- 172.21.15.178:0/2623473597 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f381802ff60 0x7f3818032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f381c0096a0 tx=0x7f381c006b40).stop 2021-11-20T06:00:24.720 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.733+0000 7f38311d1700 1 -- 172.21.15.178:0/2623473597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f382c0f6510 msgr2=0x7f382c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:24.720 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.733+0000 7f38311d1700 1 --2- 172.21.15.178:0/2623473597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f382c0f6510 0x7f382c06b9a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3814013510 tx=0x7f381400b040).stop 2021-11-20T06:00:24.720 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.733+0000 7f38311d1700 1 -- 172.21.15.178:0/2623473597 shutdown_connections 2021-11-20T06:00:24.720 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.733+0000 7f38311d1700 1 --2- 172.21.15.178:0/2623473597 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f381802ff60 0x7f3818032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:24.721 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.733+0000 7f38311d1700 1 --2- 172.21.15.178:0/2623473597 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f382c0f6510 0x7f382c06b9a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:24.721 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.733+0000 7f38311d1700 1 -- 172.21.15.178:0/2623473597 >> 172.21.15.178:0/2623473597 conn(0x7f382c0f1c70 msgr2=0x7f382c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:24.721 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.733+0000 7f38311d1700 1 -- 172.21.15.178:0/2623473597 shutdown_connections 2021-11-20T06:00:24.721 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:24.733+0000 7f38311d1700 1 -- 172.21.15.178:0/2623473597 wait complete. 2021-11-20T06:00:24.722 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2021-11-20T06:00:25.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:24 smithi139 conmon[33501]: audit 2021-11-20T06:00:24.731803+0000 mon.smithi139 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.178:0/2623473597' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T06:00:25.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:25 smithi178 systemd[1]: Starting Ceph mon.smithi178 for d819f2cc-49c6-11ec-8c2c-001a4aab830c... 2021-11-20T06:00:26.235 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-20T06:00:26.236 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-20T06:00:26.264 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:25 smithi178 podman[36558]: 2021-11-20T06:00:26.264 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.177+0000 7fb501cd3700 0 set uid:gid to 167:167 (ceph:ceph) 2021-11-20T06:00:26.265 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.177+0000 7fb501cd3700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2021-11-20T06:00:26.265 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.177+0000 7fb501cd3700 0 pidfile_write: ignore empty --pid-file 2021-11-20T06:00:26.265 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.181+0000 7fb501cd3700 0 load: jerasure load: lrc load: isa 2021-11-20T06:00:26.265 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: RocksDB version: 6.8.1 2021-11-20T06:00:26.266 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.266 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.266 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-11-20T06:00:26.266 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Compile date May 13 2021 2021-11-20T06:00:26.267 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: DB SUMMARY 2021-11-20T06:00:26.267 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.267 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: CURRENT file: CURRENT 2021-11-20T06:00:26.267 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.268 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: IDENTITY file: IDENTITY 2021-11-20T06:00:26.268 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.268 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2021-11-20T06:00:26.269 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.269 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi178/store.db dir, Total Num: 0, files: 2021-11-20T06:00:26.269 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.270 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi178/store.db: 000003.log size: 511 ; 2021-11-20T06:00:26.270 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.270 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.error_if_exists: 0 2021-11-20T06:00:26.270 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.create_if_missing: 0 2021-11-20T06:00:26.271 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.paranoid_checks: 1 2021-11-20T06:00:26.271 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.env: 0x562acb7f41c0 2021-11-20T06:00:26.271 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.fs: Posix File System 2021-11-20T06:00:26.271 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.info_log: 0x562acdf5f960 2021-11-20T06:00:26.271 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_file_opening_threads: 16 2021-11-20T06:00:26.272 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.statistics: (nil) 2021-11-20T06:00:26.272 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.use_fsync: 0 2021-11-20T06:00:26.272 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.272 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_log_file_size: 0 2021-11-20T06:00:26.272 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-11-20T06:00:26.273 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-11-20T06:00:26.273 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.keep_log_file_num: 1000 2021-11-20T06:00:26.274 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.recycle_log_file_num: 0 2021-11-20T06:00:26.274 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.allow_fallocate: 1 2021-11-20T06:00:26.274 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.allow_mmap_reads: 0 2021-11-20T06:00:26.274 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.allow_mmap_writes: 0 2021-11-20T06:00:26.275 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.use_direct_reads: 0 2021-11-20T06:00:26.275 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-11-20T06:00:26.275 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.create_missing_column_families: 0 2021-11-20T06:00:26.275 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.db_log_dir: 2021-11-20T06:00:26.275 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi178/store.db 2021-11-20T06:00:26.276 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.276 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-11-20T06:00:26.276 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_subcompactions: 1 2021-11-20T06:00:26.276 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_background_flushes: -1 2021-11-20T06:00:26.277 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-11-20T06:00:26.277 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-11-20T06:00:26.277 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-11-20T06:00:26.277 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-11-20T06:00:26.277 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-11-20T06:00:26.278 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.advise_random_on_open: 1 2021-11-20T06:00:26.278 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.db_write_buffer_size: 0 2021-11-20T06:00:26.278 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.write_buffer_manager: 0x562acdf65fb0 2021-11-20T06:00:26.278 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-11-20T06:00:26.278 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-11-20T06:00:26.279 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-11-20T06:00:26.279 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-11-20T06:00:26.280 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.rate_limiter: (nil) 2021-11-20T06:00:26.280 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-11-20T06:00:26.280 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.wal_recovery_mode: 2 2021-11-20T06:00:26.280 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.enable_thread_tracking: 0 2021-11-20T06:00:26.281 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.enable_pipelined_write: 0 2021-11-20T06:00:26.281 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.unordered_write: 0 2021-11-20T06:00:26.281 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-11-20T06:00:26.281 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-11-20T06:00:26.282 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-11-20T06:00:26.282 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-11-20T06:00:26.282 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.row_cache: None 2021-11-20T06:00:26.282 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.wal_filter: None 2021-11-20T06:00:26.283 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-11-20T06:00:26.283 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.allow_ingest_behind: 0 2021-11-20T06:00:26.283 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.preserve_deletes: 0 2021-11-20T06:00:26.284 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.two_write_queues: 0 2021-11-20T06:00:26.284 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.manual_wal_flush: 0 2021-11-20T06:00:26.284 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.285 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.atomic_flush: 0 2021-11-20T06:00:26.285 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-11-20T06:00:26.285 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-11-20T06:00:26.285 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-11-20T06:00:26.285 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.log_readahead_size: 0 2021-11-20T06:00:26.286 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-11-20T06:00:26.286 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_background_jobs: 2 2021-11-20T06:00:26.286 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_background_compactions: -1 2021-11-20T06:00:26.286 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-11-20T06:00:26.286 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-11-20T06:00:26.288 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:25 smithi139 conmon[33501]: cluster 2021-11-20T06:00:24.178266+0000 mgr.smithi139.dplyxn (mgr.14162) 38 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:26.288 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:25 smithi139 conmon[33501]: 2021-11-20T06:00:26.290 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-11-20T06:00:26.290 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_total_wal_size: 0 2021-11-20T06:00:26.291 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-11-20T06:00:26.291 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-11-20T06:00:26.291 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-11-20T06:00:26.291 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-11-20T06:00:26.292 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_open_files: -1 2021-11-20T06:00:26.292 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.bytes_per_sync: 0 2021-11-20T06:00:26.292 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-11-20T06:00:26.292 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-11-20T06:00:26.292 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compaction_readahead_size: 0 2021-11-20T06:00:26.293 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Compression algorithms supported: 2021-11-20T06:00:26.293 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-11-20T06:00:26.293 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: kZSTD supported: 0 2021-11-20T06:00:26.293 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: kXpressCompression supported: 0 2021-11-20T06:00:26.294 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: kLZ4HCCompression supported: 1 2021-11-20T06:00:26.295 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: kLZ4Compression supported: 1 2021-11-20T06:00:26.295 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: kBZip2Compression supported: 0 2021-11-20T06:00:26.295 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: kZlibCompression supported: 1 2021-11-20T06:00:26.295 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: kSnappyCompression supported: 1 2021-11-20T06:00:26.295 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-11-20T06:00:26.296 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi178/store.db/MANIFEST-000001 2021-11-20T06:00:26.296 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.296 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-11-20T06:00:26.296 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.297 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-11-20T06:00:26.297 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.merge_operator: 2021-11-20T06:00:26.297 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compaction_filter: None 2021-11-20T06:00:26.297 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compaction_filter_factory: None 2021-11-20T06:00:26.298 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-11-20T06:00:26.298 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-11-20T06:00:26.298 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.299 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562acde76128) 2021-11-20T06:00:26.299 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: cache_index_and_filter_blocks: 1 2021-11-20T06:00:26.299 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: cache_index_and_filter_blocks_with_high_priority: 0 2021-11-20T06:00:26.299 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-11-20T06:00:26.299 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: pin_top_level_index_and_filter: 1 2021-11-20T06:00:26.300 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: index_type: 0 2021-11-20T06:00:26.300 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: data_block_index_type: 0 2021-11-20T06:00:26.301 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: index_shortening: 1 2021-11-20T06:00:26.301 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: data_block_hash_table_util_ratio: 0.750000 2021-11-20T06:00:26.301 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: hash_index_allow_collision: 1 2021-11-20T06:00:26.301 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: checksum: 1 2021-11-20T06:00:26.302 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: no_block_cache: 0 2021-11-20T06:00:26.302 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: block_cache: 0x562acdeacf10 2021-11-20T06:00:26.302 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: block_cache_name: BinnedLRUCache 2021-11-20T06:00:26.302 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: block_cache_options: 2021-11-20T06:00:26.302 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: capacity : 536870912 2021-11-20T06:00:26.303 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: num_shard_bits : 4 2021-11-20T06:00:26.303 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: strict_capacity_limit : 0 2021-11-20T06:00:26.303 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: high_pri_pool_ratio: 0.000 2021-11-20T06:00:26.303 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: block_cache_compressed: (nil) 2021-11-20T06:00:26.303 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: persistent_cache: (nil) 2021-11-20T06:00:26.304 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: block_size: 4096 2021-11-20T06:00:26.304 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: block_size_deviation: 10 2021-11-20T06:00:26.305 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: block_restart_interval: 16 2021-11-20T06:00:26.305 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: index_block_restart_interval: 1 2021-11-20T06:00:26.305 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: metadata_block_size: 4096 2021-11-20T06:00:26.305 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: partition_filters: 0 2021-11-20T06:00:26.305 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: use_delta_encoding: 1 2021-11-20T06:00:26.306 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: filter_policy: rocksdb.BuiltinBloomFilter 2021-11-20T06:00:26.306 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: whole_key_filtering: 1 2021-11-20T06:00:26.306 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: verify_compression: 0 2021-11-20T06:00:26.306 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: read_amp_bytes_per_bit: 0 2021-11-20T06:00:26.307 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: format_version: 2 2021-11-20T06:00:26.307 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: enable_index_compression: 1 2021-11-20T06:00:26.307 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: block_align: 0 2021-11-20T06:00:26.307 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.307 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.write_buffer_size: 33554432 2021-11-20T06:00:26.308 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_write_buffer_number: 2 2021-11-20T06:00:26.308 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compression: NoCompression 2021-11-20T06:00:26.308 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.bottommost_compression: Disabled 2021-11-20T06:00:26.308 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.prefix_extractor: nullptr 2021-11-20T06:00:26.308 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-11-20T06:00:26.309 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.num_levels: 7 2021-11-20T06:00:26.309 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-11-20T06:00:26.309 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-11-20T06:00:26.309 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-11-20T06:00:26.310 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-11-20T06:00:26.310 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-11-20T06:00:26.310 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-11-20T06:00:26.310 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-11-20T06:00:26.310 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-11-20T06:00:26.311 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-11-20T06:00:26.311 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-11-20T06:00:26.312 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compression_opts.level: 32767 2021-11-20T06:00:26.312 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compression_opts.strategy: 0 2021-11-20T06:00:26.312 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-11-20T06:00:26.313 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-11-20T06:00:26.313 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compression_opts.enabled: false 2021-11-20T06:00:26.313 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-11-20T06:00:26.313 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-11-20T06:00:26.314 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-11-20T06:00:26.314 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.target_file_size_base: 67108864 2021-11-20T06:00:26.315 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-11-20T06:00:26.315 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-11-20T06:00:26.316 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-11-20T06:00:26.316 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-11-20T06:00:26.316 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-11-20T06:00:26.316 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-11-20T06:00:26.316 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-11-20T06:00:26.317 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-11-20T06:00:26.317 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-11-20T06:00:26.317 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-11-20T06:00:26.317 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-11-20T06:00:26.317 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-11-20T06:00:26.318 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-11-20T06:00:26.318 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.arena_block_size: 4194304 2021-11-20T06:00:26.318 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-11-20T06:00:26.318 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-11-20T06:00:26.319 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-11-20T06:00:26.319 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.disable_auto_compactions: 0 2021-11-20T06:00:26.319 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-11-20T06:00:26.319 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-11-20T06:00:26.320 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-11-20T06:00:26.320 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-11-20T06:00:26.320 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-11-20T06:00:26.321 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-11-20T06:00:26.321 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-11-20T06:00:26.321 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-11-20T06:00:26.323 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-11-20T06:00:26.323 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-11-20T06:00:26.324 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.table_properties_collectors: 2021-11-20T06:00:26.324 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.inplace_update_support: 0 2021-11-20T06:00:26.324 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-11-20T06:00:26.324 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-11-20T06:00:26.324 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-11-20T06:00:26.325 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-11-20T06:00:26.326 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.bloom_locality: 0 2021-11-20T06:00:26.326 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.max_successive_merges: 0 2021-11-20T06:00:26.326 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-11-20T06:00:26.327 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.paranoid_file_checks: 0 2021-11-20T06:00:26.327 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.force_consistency_checks: 0 2021-11-20T06:00:26.328 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.report_bg_io_stats: 0 2021-11-20T06:00:26.328 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.ttl: 2592000 2021-11-20T06:00:26.328 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-11-20T06:00:26.328 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.329 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi178/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-11-20T06:00:26.329 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.329 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.182+0000 7fb501cd3700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-11-20T06:00:26.329 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.330 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.183+0000 7fb501cd3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637388026183870, "job": 1, "event": "recovery_started", "log_files": [3]} 2021-11-20T06:00:26.330 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.330 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.183+0000 7fb501cd3700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2021-11-20T06:00:26.330 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.183+0000 7fb501cd3700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-11-20T06:00:26.331 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.183+0000 7fb501cd3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637388026184486, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637388026, "oldest_key_time": 3, "file_creation_time": 0}} 2021-11-20T06:00:26.331 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.183+0000 7fb501cd3700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2021-11-20T06:00:26.331 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.331 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.184+0000 7fb501cd3700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637388026185009, "job": 1, "event": "recovery_finished"} 2021-11-20T06:00:26.331 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.185+0000 7fb501cd3700 4 rocksdb: DB pointer 0x562acdf71800 2021-11-20T06:00:26.332 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.185+0000 7fb4e8aeb700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-11-20T06:00:26.332 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.185+0000 7fb4e8aeb700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-11-20T06:00:26.332 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: ** DB Stats ** 2021-11-20T06:00:26.332 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T06:00:26.332 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-11-20T06:00:26.333 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-11-20T06:00:26.333 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T06:00:26.333 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-11-20T06:00:26.333 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-11-20T06:00:26.334 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-20T06:00:26.334 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.334 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: ** Compaction Stats [default] ** 2021-11-20T06:00:26.334 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T06:00:26.334 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T06:00:26.335 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.4 0.00 0.00 1 0.001 0 0 2021-11-20T06:00:26.335 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.4 0.00 0.00 1 0.001 0 0 2021-11-20T06:00:26.335 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.4 0.00 0.00 1 0.001 0 0 2021-11-20T06:00:26.335 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.336 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: ** Compaction Stats [default] ** 2021-11-20T06:00:26.336 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T06:00:26.336 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T06:00:26.336 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.4 0.00 0.00 1 0.001 0 0 2021-11-20T06:00:26.336 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T06:00:26.337 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T06:00:26.337 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T06:00:26.337 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T06:00:26.337 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T06:00:26.337 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T06:00:26.338 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Cumulative compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T06:00:26.338 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 bash[36483]: 3dba80a538b50175ce78587279c5f859415be0c3a6e508c9576e93bc96b2b4e6 2021-11-20T06:00:26.338 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Interval compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T06:00:26.338 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T06:00:26.339 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.339 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T06:00:26.339 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.339 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: ** Compaction Stats [default] ** 2021-11-20T06:00:26.340 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T06:00:26.340 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T06:00:26.340 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.4 0.00 0.00 1 0.001 0 0 2021-11-20T06:00:26.340 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.4 0.00 0.00 1 0.001 0 0 2021-11-20T06:00:26.340 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-20T06:00:26.341 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.341 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: ** Compaction Stats [default] ** 2021-11-20T06:00:26.341 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-20T06:00:26.341 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-20T06:00:26.342 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.4 0.00 0.00 1 0.001 0 0 2021-11-20T06:00:26.342 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-20T06:00:26.342 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-20T06:00:26.342 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-20T06:00:26.342 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: AddFile(Total Files): cumulative 0, interval 0 2021-11-20T06:00:26.343 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-20T06:00:26.343 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: AddFile(Keys): cumulative 0, interval 0 2021-11-20T06:00:26.343 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Cumulative compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T06:00:26.343 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-20T06:00:26.343 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-20T06:00:26.344 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.344 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: ** File Read Latency Histogram By Level [default] ** 2021-11-20T06:00:26.344 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.344 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.185+0000 7fb501cd3700 0 mon.smithi178 does not exist in monmap, will attempt to join an existing cluster 2021-11-20T06:00:26.345 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.185+0000 7fb501cd3700 0 using public_addr v2:172.21.15.178:0/0 -> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2021-11-20T06:00:26.345 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.186+0000 7fb501cd3700 0 starting mon.smithi178 rank -1 at public addrs [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] at bind addrs [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi178 fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T06:00:26.345 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.187+0000 7fb501cd3700 1 mon.smithi178@-1(???) e0 preinit fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T06:00:26.345 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.210+0000 7fb4eb2f0700 0 mon.smithi178@-1(synchronizing).mds e1 new map 2021-11-20T06:00:26.345 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.210+0000 7fb4eb2f0700 0 mon.smithi178@-1(synchronizing).mds e1 print_map 2021-11-20T06:00:26.346 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: e1 2021-11-20T06:00:26.346 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T06:00:26.346 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T06:00:26.346 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: legacy client fscid: -1 2021-11-20T06:00:26.347 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.347 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: No filesystems configured 2021-11-20T06:00:26.347 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.347 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.210+0000 7fb4eb2f0700 1 mon.smithi178@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2021-11-20T06:00:26.347 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.210+0000 7fb4eb2f0700 1 mon.smithi178@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2021-11-20T06:00:26.348 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.210+0000 7fb4eb2f0700 1 mon.smithi178@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2021-11-20T06:00:26.348 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.210+0000 7fb4eb2f0700 1 mon.smithi178@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2021-11-20T06:00:26.348 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.210+0000 7fb4eb2f0700 1 mon.smithi178@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2021-11-20T06:00:26.348 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.210+0000 7fb4eb2f0700 1 mon.smithi178@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2021-11-20T06:00:26.348 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.210+0000 7fb4eb2f0700 1 mon.smithi178@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2021-11-20T06:00:26.349 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.210+0000 7fb4eb2f0700 0 mon.smithi178@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2021-11-20T06:00:26.349 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.210+0000 7fb4eb2f0700 0 mon.smithi178@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T06:00:26.349 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.210+0000 7fb4eb2f0700 0 mon.smithi178@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T06:00:26.349 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.210+0000 7fb4eb2f0700 0 mon.smithi178@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2021-11-20T06:00:26.350 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: cluster 2021-11- 2021-11-20T06:00:26.350 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 20T06:00:24.178266+0000 mgr.smithi139.dplyxn (mgr.14162) 38 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:26.350 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.350 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.210+0000 7fb4eb2f0700 1 mon.smithi178@-1(synchronizing).paxosservice(auth 1..7) refresh upgraded, format 0 -> 3 2021-11-20T06:00:26.350 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.212+0000 7fb4eb2f0700 20 expand_channel_meta expand map: {default=false} 2021-11-20T06:00:26.351 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.212+0000 7fb4eb2f0700 20 expand_channel_meta from 'false' to 'false' 2021-11-20T06:00:26.351 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.212+0000 7fb4eb2f0700 20 expand_channel_meta expanded map: {default=false} 2021-11-20T06:00:26.351 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.212+0000 7fb4eb2f0700 20 expand_channel_meta expand map: {default=info} 2021-11-20T06:00:26.351 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.212+0000 7fb4eb2f0700 20 expand_channel_meta from 'info' to 'info' 2021-11-20T06:00:26.351 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.212+0000 7fb4eb2f0700 20 expand_channel_meta expanded map: {default=info} 2021-11-20T06:00:26.352 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.212+0000 7fb4eb2f0700 20 expand_channel_meta expand map: {default=daemon} 2021-11-20T06:00:26.352 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.212+0000 7fb4eb2f0700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-11-20T06:00:26.352 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.212+0000 7fb4eb2f0700 20 expand_channel_meta expanded map: {default=daemon} 2021-11-20T06:00:26.352 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.212+0000 7fb4eb2f0700 20 expand_channel_meta expand map: {default=debug} 2021-11-20T06:00:26.353 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.212+0000 7fb4eb2f0700 20 expand_channel_meta from 'debug' to 'debug' 2021-11-20T06:00:26.353 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.212+0000 7fb4eb2f0700 20 expand_channel_meta expanded map: {default=debug} 2021-11-20T06:00:26.353 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: 2021-11-20T06:00:26.353 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 conmon[36600]: debug 2021-11-20T06:00:26.212+0000 7fb4eb2f0700 10 mon.smithi178@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2021-11-20T06:00:26.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:26 smithi178 systemd[1]: Started Ceph mon.smithi178 for d819f2cc-49c6-11ec-8c2c-001a4aab830c. 2021-11-20T06:00:31.411 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.419+0000 7f87ad8b1700 1 -- 172.21.15.178:0/2094210446 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8798003250 msgr2=0x7f87980056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:31.411 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.419+0000 7f87ad8b1700 1 --2- 172.21.15.178:0/2094210446 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8798003250 0x7f87980056e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f87a805e720 tx=0x7f879c00b040).stop 2021-11-20T06:00:31.411 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.419+0000 7f87ad8b1700 1 -- 172.21.15.178:0/2094210446 shutdown_connections 2021-11-20T06:00:31.412 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.419+0000 7f87ad8b1700 1 --2- 172.21.15.178:0/2094210446 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8798003250 0x7f87980056e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:31.412 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.419+0000 7f87ad8b1700 1 -- 172.21.15.178:0/2094210446 >> 172.21.15.178:0/2094210446 conn(0x7f87a80f0120 msgr2=0x7f87a80f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:31.412 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.419+0000 7f87ad8b1700 1 -- 172.21.15.178:0/2094210446 shutdown_connections 2021-11-20T06:00:31.412 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.419+0000 7f87ad8b1700 1 -- 172.21.15.178:0/2094210446 wait complete. 2021-11-20T06:00:31.413 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.420+0000 7f87ad8b1700 1 Processor -- start 2021-11-20T06:00:31.413 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.420+0000 7f87ad8b1700 1 -- start start 2021-11-20T06:00:31.414 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.420+0000 7f87ad8b1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8798003250 0x7f87a8106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:31.414 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.420+0000 7f87ad8b1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87a8106a10 0x7f87a8107900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:31.414 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.420+0000 7f87ad8b1700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a8106f70 con 0x7f8798003250 2021-11-20T06:00:31.414 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87ad8b1700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a81070b0 con 0x7f87a8106a10 2021-11-20T06:00:31.415 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87a7fff700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87a8106a10 0x7f87a8107900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:31.415 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87ac8af700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8798003250 0x7f87a8106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:31.415 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87a7fff700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87a8106a10 0x7f87a8107900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:50092/0 (socket says 172.21.15.178:50092) 2021-11-20T06:00:31.415 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87a7fff700 1 -- 172.21.15.178:0/2179024123 learned_addr learned my addr 172.21.15.178:0/2179024123 (peer_addr_for_me v2:172.21.15.178:0/0) 2021-11-20T06:00:31.415 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87a7fff700 1 -- 172.21.15.178:0/2179024123 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87a8106a10 msgr2=0x7f87a8107900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-11-20T06:00:31.416 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87a7fff700 1 -- 172.21.15.178:0/2179024123 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87a8106a10 msgr2=0x7f87a8107900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-11-20T06:00:31.416 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87a7fff700 1 --2- 172.21.15.178:0/2179024123 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87a8106a10 0x7f87a8107900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T06:00:31.416 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87a7fff700 1 --2- 172.21.15.178:0/2179024123 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87a8106a10 0x7f87a8107900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T06:00:31.416 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87ac8af700 1 -- 172.21.15.178:0/2179024123 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87a8106a10 msgr2=0x7f87a8107900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-20T06:00:31.417 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87ac8af700 1 --2- 172.21.15.178:0/2179024123 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87a8106a10 0x7f87a8107900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:31.417 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87ac8af700 1 -- 172.21.15.178:0/2179024123 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879c005040 con 0x7f8798003250 2021-11-20T06:00:31.417 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87ac8af700 1 --2- 172.21.15.178:0/2179024123 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8798003250 0x7f87a8106520 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f87a805e720 tx=0x7f879c002040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:31.417 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87a57fa700 1 -- 172.21.15.178:0/2179024123 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f879c017840 con 0x7f8798003250 2021-11-20T06:00:31.417 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87a57fa700 1 -- 172.21.15.178:0/2179024123 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f879c017e50 con 0x7f8798003250 2021-11-20T06:00:31.418 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87a57fa700 1 -- 172.21.15.178:0/2179024123 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f879c021da0 con 0x7f8798003250 2021-11-20T06:00:31.418 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87ad8b1700 1 -- 172.21.15.178:0/2179024123 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a8107eb0 con 0x7f8798003250 2021-11-20T06:00:31.418 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.421+0000 7f87ad8b1700 1 -- 172.21.15.178:0/2179024123 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a8108470 con 0x7f8798003250 2021-11-20T06:00:31.418 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.422+0000 7f87ad8b1700 1 -- 172.21.15.178:0/2179024123 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a8059100 con 0x7f8798003250 2021-11-20T06:00:31.418 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.423+0000 7f87a57fa700 1 -- 172.21.15.178:0/2179024123 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f879c031070 con 0x7f8798003250 2021-11-20T06:00:31.419 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.423+0000 7f87a57fa700 1 --2- 172.21.15.178:0/2179024123 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f878c05c3b0 0x7f878c05e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:31.419 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.423+0000 7f87a57fa700 1 -- 172.21.15.178:0/2179024123 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f879c077f80 con 0x7f8798003250 2021-11-20T06:00:31.419 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.423+0000 7f87a7fff700 1 --2- 172.21.15.178:0/2179024123 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f878c05c3b0 0x7f878c05e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:31.419 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.424+0000 7f87a7fff700 1 --2- 172.21.15.178:0/2179024123 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f878c05c3b0 0x7f878c05e870 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f8794006cf0 tx=0x7f8794008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:31.420 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.427+0000 7f87a57fa700 1 -- 172.21.15.178:0/2179024123 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f879c04bc50 con 0x7f8798003250 2021-11-20T06:00:31.636 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.648+0000 7f87ad8b1700 1 -- 172.21.15.178:0/2179024123 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87a8045ad0 con 0x7f8798003250 2021-11-20T06:00:31.636 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.649+0000 7f87a57fa700 1 -- 172.21.15.178:0/2179024123 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+972 (secure 0 0 0) 0x7f879c025070 con 0x7f8798003250 2021-11-20T06:00:31.637 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: cluster 2021-11-20T06:00:26.178550+0000 mgr.smithi139.dplyxn (mgr.14162) 39 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:31.638 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: audit 2021-11-20T06:00:26.218848+0000 mon.smithi139 (mon.0) 158 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi139"}]: dispatch 2021-11-20T06:00:31.638 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: cluster 2021-11-20T06:00:26.219071+0000 mon.smithi139 (mon.0) 159 : cluster [INF] mon.smithi139 calling monitor election 2021-11-20T06:00:31.639 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: audit 2021-11-20 2021-11-20T06:00:31.640 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: T06:00:26.231375+0000 mon.smithi139 (mon.0) 160 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi178"}]: dispatch 2021-11-20T06:00:31.640 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: audit 2021-11-20T06:00:26.302277+0000 2021-11-20T06:00:31.641 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: mon.smithi139 (mon.0) 161 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:00:31.642 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: audit 2021-11-20 2021-11-20T06:00:31.643 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: T06:00:26.311473+0000 mon.smithi139 (mon.0) 162 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:00:31.643 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: audit 2021-11-20T06:00:27.215021+0000 mon.smithi139 ( 2021-11-20T06:00:31.644 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: mon.0) 163 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi178"}]: dispatch 2021-11-20T06:00:31.644 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: audit 2021-11-20T06:00:28.006332+0000 mon.smithi139 (mon. 2021-11-20T06:00:31.645 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: 0) 164 : audit [DBG] from='mgr.? 172.21.15.178:0/2664797593' entity='mgr.smithi178.zxxdkz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi178.zxxdkz/crt"}]: dispatch 2021-11-20T06:00:31.645 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: cluster 2021-11-20T06:00:28.178936+0000 mgr.smithi139.dplyxn (mgr 2021-11-20T06:00:31.645 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: .14162) 40 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:31.646 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: audit 2021-11-20T06:00:28.215130+0000 mon.smithi139 (mon 2021-11-20T06:00:31.646 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: .0) 165 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi178"}]: dispatch 2021-11-20T06:00:31.646 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: cluster 2021-11-20T06:00:28.216334+0000 mon.smithi178 (mon.1) 1 : cluster [INF] 2021-11-20T06:00:31.647 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: mon.smithi178 calling monitor election 2021-11-20T06:00:31.647 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: audit 2021-11-20T06 2021-11-20T06:00:31.647 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: :00:29.215289+0000 mon.smithi139 (mon.0) 166 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi178"}]: dispatch 2021-11-20T06:00:31.648 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: cluster 2021-11-20T06:00:30.179263+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:00:31.648 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: ) 41 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:31.648 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: audit 2021-11-20T06:00:31.648 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: 2021-11-20T06:00:30.215393+0000 mon.smithi139 (mon.0) 167 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi178"}]: dispatch 2021-11-20T06:00:31.649 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: 2021-11-20T06:00:31.649 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: audit 2021-11-20T06:00 2021-11-20T06:00:31.652 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-11-20T06:00:31.652 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":2,"fsid":"d819f2cc-49c6-11ec-8c2c-001a4aab830c","modified":"2021-11-20T06:00:26.215375Z","created":"2021-11-20T05:58:31.415306Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi139","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi178","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:3300","nonce":0},{"type":"v1","addr":"172.21.15.178:6789","nonce":0}]},"addr":"172.21.15.178:6789/0","public_addr":"172.21.15.178:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2021-11-20T06:00:31.653 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: :31.215545+0000 mon.smithi139 (mon.0) 168 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi178"}]: dispatch 2021-11-20T06:00:31.653 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: cluster 2021-11-20T06:00:31.244797+0000 mon.smithi139 (mon 2021-11-20T06:00:31.653 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: .0) 169 : cluster [INF] mon.smithi139 is new leader, mons smithi139,smithi178 in quorum (ranks 0,1) 2021-11-20T06:00:31.653 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: cluster 2021-11-20T06:00:31.247818+0000 mon.smithi139 (mon.0 2021-11-20T06:00:31.654 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: ) 170 : cluster [DBG] monmap e2: 2 mons at {smithi139=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],smithi178=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} 2021-11-20T06:00:31.654 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: cluster 2021-11-20T06:00:31.247871+0000 mon.smithi139 (mon 2021-11-20T06:00:31.654 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: .0) 171 : cluster [DBG] fsmap 2021-11-20T06:00:31.654 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: cluster 2021-11-20T06:00:31.247904+0000 mon.smithi139 (mon 2021-11-20T06:00:31.655 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: .0) 172 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2021-11-20T06:00:31.655 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: cluster 2021-11-20T06:00:31.655 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: 2021-11-20T06:00:31.248133+0000 mon.smithi139 (mon.0) 173 : cluster [DBG] 2021-11-20T06:00:31.655 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: mgrmap e16: smithi139.dplyxn(active, since 65s) 2021-11-20T06:00:31.655 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: cluster 2021-11-20T 2021-11-20T06:00:31.656 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: 06:00:31.249160+0000 mon.smithi139 (mon.0) 174 : cluster [DBG] Standby manager daemon smithi178.zxxdkz started 2021-11-20T06:00:31.656 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: audit 2021-11-20T06:00:31.250405+0000 2021-11-20T06:00:31.656 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: mon.smithi139 (mon.0) 175 : audit [DBG] from='mgr.? 172.21.15.178:0/2664797593' entity='mgr.smithi178.zxxdkz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-11-20T06:00:31.656 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: cluster 2021-11-20T06:00:31.251904+0000 2021-11-20T06:00:31.657 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: mon.smithi139 (mon.0) 176 : cluster [INF] overall HEALTH_OK 2021-11-20T06:00:31.657 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: audit 2021-11-20T06:00:31.252318+0000 2021-11-20T06:00:31.657 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: mon.smithi139 (mon.0) 177 : audit [DBG] from='mgr.? 172.21.15.178:0/2664797593' entity='mgr.smithi178.zxxdkz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi178.zxxdkz/key"}]: dispatch 2021-11-20T06:00:31.658 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: audit 2021-11-20T06:00:31.253999+0000 mon.smithi139 ( 2021-11-20T06:00:31.658 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: mon.0) 178 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:31.658 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: audit 2021-11-20T06:00:31.254157+0000 mon.smithi139 (mon. 2021-11-20T06:00:31.658 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:31 smithi178 conmon[36600]: 0) 179 : audit [DBG] from='mgr.? 172.21.15.178:0/2664797593' entity='mgr.smithi178.zxxdkz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-11-20T06:00:31.659 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.652+0000 7f87ad8b1700 1 -- 172.21.15.178:0/2179024123 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f878c05c3b0 msgr2=0x7f878c05e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:31.659 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.652+0000 7f87ad8b1700 1 --2- 172.21.15.178:0/2179024123 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f878c05c3b0 0x7f878c05e870 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f8794006cf0 tx=0x7f8794008040).stop 2021-11-20T06:00:31.660 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.652+0000 7f87ad8b1700 1 -- 172.21.15.178:0/2179024123 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8798003250 msgr2=0x7f87a8106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:31.660 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.652+0000 7f87ad8b1700 1 --2- 172.21.15.178:0/2179024123 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8798003250 0x7f87a8106520 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f87a805e720 tx=0x7f879c002040).stop 2021-11-20T06:00:31.660 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.652+0000 7f87ad8b1700 1 -- 172.21.15.178:0/2179024123 shutdown_connections 2021-11-20T06:00:31.660 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.652+0000 7f87ad8b1700 1 --2- 172.21.15.178:0/2179024123 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f878c05c3b0 0x7f878c05e870 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:31.661 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.652+0000 7f87ad8b1700 1 --2- 172.21.15.178:0/2179024123 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8798003250 0x7f87a8106520 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:31.661 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.652+0000 7f87ad8b1700 1 --2- 172.21.15.178:0/2179024123 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87a8106a10 0x7f87a8107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:31.661 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.652+0000 7f87ad8b1700 1 -- 172.21.15.178:0/2179024123 >> 172.21.15.178:0/2179024123 conn(0x7f87a80f0120 msgr2=0x7f87a80fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:31.661 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.652+0000 7f87ad8b1700 1 -- 172.21.15.178:0/2179024123 shutdown_connections 2021-11-20T06:00:31.662 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:00:31.652+0000 7f87ad8b1700 1 -- 172.21.15.178:0/2179024123 wait complete. 2021-11-20T06:00:31.662 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 2 2021-11-20T06:00:31.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: cluster 2021-11-20T06:00:26.178550+0000 mgr.smithi139.dplyxn (mgr.14162) 39 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:31.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: audit 2021-11 2021-11-20T06:00:31.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: -20T06:00:26.218848+0000 mon.smithi139 (mon.0) 158 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi139"}]: dispatch 2021-11-20T06:00:31.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: cluster 2021-11-20T06:00:26.219071+0000 mon.smithi139 (mon.0) 159 : cluster [INF] mon.smithi139 calling monitor election 2021-11-20T06:00:31.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: audit 2021-11-20T06:00:26.231375 2021-11-20T06:00:31.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 160 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi178"}]: dispatch 2021-11-20T06:00:31.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: audit 2021-11-20T06:00:26.302277+0000 mon.smithi139 (mon.0) 161 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:00:31.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: audit 2021-11-20T06:00:26.311473+0000 mon.smithi139 (mon.0) 162 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:00:31.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:00:31.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 20T06:00:27.215021+0000 mon.smithi139 (mon. 2021-11-20T06:00:31.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 0) 163 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi178"}]: dispatch 2021-11-20T06:00:31.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:00:31.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 20T06:00:28.006332+0000 mon.smithi139 (mon 2021-11-20T06:00:31.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: .0) 164 : audit [DBG] from='mgr.? 172.21.15.178:0/2664797593' entity='mgr.smithi178.zxxdkz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi178.zxxdkz/crt"}]: dispatch 2021-11-20T06:00:31.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: cluster 2021-11- 2021-11-20T06:00:31.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 20T06:00:28.178936+0000 mgr.smithi139.dplyxn (mgr 2021-11-20T06:00:31.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: .14162) 40 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:31.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: audit 2021-11 2021-11-20T06:00:31.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: -20T06:00:28.215130+0000 mon.smithi139 ( 2021-11-20T06:00:31.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: mon.0) 165 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi178"}]: dispatch 2021-11-20T06:00:31.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: cluster 2021- 2021-11-20T06:00:31.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 11-20T06:00:28.216334+0000 mon.smithi178 2021-11-20T06:00:31.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: (mon.1) 1 : cluster [INF] mon.smithi178 calling monitor election 2021-11-20T06:00:31.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: audit 2021 2021-11-20T06:00:31.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: -11-20T06:00:29.215289+0000 2021-11-20T06:00:31.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: mon.smithi139 (mon.0) 166 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi178"}]: dispatch 2021-11-20T06:00:31.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 2021-11-20T06:00:31.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: cluster 2021-11-20T06:00:30 2021-11-20T06:00:31.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: .179263+0000 mgr.smithi139.dplyxn (mgr.14162) 41 : cluster 2021-11-20T06:00:31.786 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:31.786 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: audit 2021-11-20T06 2021-11-20T06:00:31.786 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: :00:30.215393+0000 mon.smithi139 (mon.0) 167 2021-11-20T06:00:31.786 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi178"}]: dispatch 2021-11-20T06:00:31.787 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: audit 2021-11-20T06: 2021-11-20T06:00:31.787 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 00:31.215545+0000 mon.smithi139 (mon.0) 168 : audit 2021-11-20T06:00:31.787 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi178"}]: dispatch 2021-11-20T06:00:31.787 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: cluster 2021-11-20T06:00:31.244797+0000 mon.smithi139 (mon.0) 169 : cluster [INF] mon.smithi139 is new leader, mons smithi139,smithi178 in quorum (ranks 0,1) 2021-11-20T06:00:31.787 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 2021-11-20T06:00:31.788 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: cluster 2021-11-20T06:00:31.788 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 2021-11-20T06:00:31.247818+0000 mon.smithi139 ( 2021-11-20T06:00:31.788 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: mon.0) 170 : cluster [DBG] monmap e2: 2 mons at {smithi139=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],smithi178=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} 2021-11-20T06:00:31.788 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: cluster 2021- 2021-11-20T06:00:31.789 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 11-20T06:00:31.247871+0000 mon.smithi139 ( 2021-11-20T06:00:31.789 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: mon.0) 171 : cluster [DBG] fsmap 2021-11-20T06:00:31.789 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: cluster 2021- 2021-11-20T06:00:31.789 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 11-20T06:00:31.247904+0000 mon.smithi139 ( 2021-11-20T06:00:31.790 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: mon.0) 172 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2021-11-20T06:00:31.790 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: cluster 2021- 2021-11-20T06:00:31.790 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 11-20T06:00:31.248133+0000 mon.smithi139 2021-11-20T06:00:31.790 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: (mon.0) 173 : cluster [DBG] mgrmap e16: smithi139.dplyxn(active, since 65s) 2021-11-20T06:00:31.791 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: cluster 2021-11 2021-11-20T06:00:31.791 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: -20T06:00:31.249160+0000 mon.smithi139 ( 2021-11-20T06:00:31.791 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: mon.0) 174 : cluster [DBG] Standby manager daemon smithi178.zxxdkz started 2021-11-20T06:00:31.791 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: audit 2021-11 2021-11-20T06:00:31.791 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: -20T06:00:31.250405+0000 mon.smithi139 ( 2021-11-20T06:00:31.792 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: mon.0) 175 : audit [DBG] from='mgr.? 172.21.15.178:0/2664797593' entity='mgr.smithi178.zxxdkz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-11-20T06:00:31.792 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: cluster 2021 2021-11-20T06:00:31.792 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: -11-20T06:00:31.251904+0000 2021-11-20T06:00:31.793 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: mon.smithi139 (mon.0) 176 : cluster [INF] overall HEALTH_OK 2021-11-20T06:00:31.793 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: audit 2021-11-20T06:00:31.794 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 2021-11-20T06:00:31.252318+0000 2021-11-20T06:00:31.794 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: mon.smithi139 (mon.0) 177 : audit [DBG] from='mgr.? 172.21.15.178:0/2664797593' entity='mgr.smithi178.zxxdkz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi178.zxxdkz/key"}]: dispatch 2021-11-20T06:00:31.795 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 2021-11-20T06:00:31.795 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: audit 2021-11-20T06:00:31.795 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 2021-11-20T06:00:31.253999+0000 mon.smithi139 (mon 2021-11-20T06:00:31.795 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: .0) 178 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:31.796 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: audit 2021-11-20T 2021-11-20T06:00:31.796 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: 06:00:31.254157+0000 mon.smithi139 (mon.0 2021-11-20T06:00:31.796 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:31 smithi139 conmon[33501]: ) 179 : audit [DBG] from='mgr.? 172.21.15.178:0/2664797593' entity='mgr.smithi178.zxxdkz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-11-20T06:00:31.983 INFO:tasks.cephadm:Generating final ceph.conf file... 2021-11-20T06:00:31.984 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph config generate-minimal-conf 2021-11-20T06:00:32.563 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:32 smithi178 conmon[36600]: cephadm 2021-11-20T06:00:31.255074+0000 mgr.smithi139.dplyxn (mgr.14162) 42 : cephadm [INF] Deploying daemon node-exporter.smithi178 on smithi178 2021-11-20T06:00:32.563 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:32 smithi178 conmon[36600]: cluster 2021 2021-11-20T06:00:32.564 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:32 smithi178 conmon[36600]: -11-20T06:00:31.308223+0000 mon.smithi139 (mon.0) 180 : cluster [DBG] mgrmap e17: smithi139.dplyxn(active, since 65s), standbys: smithi178.zxxdkz 2021-11-20T06:00:32.564 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:32 smithi178 conmon[36600]: audit 2021-11-20T06:00:31.308389+0000 mon.smithi139 (mon.0) 181 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mgr metadata", "who": "smithi178.zxxdkz", "id": "smithi178.zxxdkz"}]: dispatch 2021-11-20T06:00:32.564 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:32 smithi178 conmon[36600]: audit 2021-11-20T06:00:31.650014+0000 mon.smithi139 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.178:0/2179024123' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T06:00:32.564 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:32 smithi178 conmon[36600]: audit 2021-11-20T06:00:32.215615+0000 mon.smithi139 (mon.0) 183 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi178"}]: dispatch 2021-11-20T06:00:32.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:32 smithi139 conmon[33501]: cephadm 2021-11-20T06:00:31.255074+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:00:32.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:32 smithi139 conmon[33501]: 42 : cephadm [INF] Deploying daemon node-exporter.smithi178 on smithi178 2021-11-20T06:00:32.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:32 smithi139 conmon[33501]: cluster 2021-11-20T06:00:31.308223+0000 mon.smithi139 (mon.0) 180 : cluster [DBG] mgrmap e17: smithi139.dplyxn(active, since 65s), standbys: smithi178.zxxdkz 2021-11-20T06:00:32.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:32 smithi139 conmon[33501]: audit 2021-11-20T06:00:31.308389+0000 mon.smithi139 (mon.0) 181 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mgr metadata", "who": "smithi178.zxxdkz", "id": "smithi178.zxxdkz"}]: dispatch 2021-11-20T06:00:32.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:32 smithi139 conmon[33501]: audit 2021-11-20T06:00:31.650014+0000 mon.smithi139 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.178:0/2179024123' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-20T06:00:32.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:32 smithi139 conmon[33501]: audit 2021-11-20T06:00:32.215615+0000 mon.smithi139 (mon.0) 183 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mon metadata", "id": "smithi178"}]: dispatch 2021-11-20T06:00:34.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.212+0000 7fa2a0c4e700 1 -- 172.21.15.139:0/427451331 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa29c0f62a0 msgr2=0x7fa29c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:34.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.212+0000 7fa2a0c4e700 1 --2- 172.21.15.139:0/427451331 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa29c0f62a0 0x7fa29c0f66c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fa28c009fb0 tx=0x7fa28c009b30).stop 2021-11-20T06:00:34.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.212+0000 7fa2a0c4e700 1 -- 172.21.15.139:0/427451331 shutdown_connections 2021-11-20T06:00:34.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.212+0000 7fa2a0c4e700 1 --2- 172.21.15.139:0/427451331 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa29c0f62a0 0x7fa29c0f66c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:34.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.212+0000 7fa2a0c4e700 1 -- 172.21.15.139:0/427451331 >> 172.21.15.139:0/427451331 conn(0x7fa29c0f1aa0 msgr2=0x7fa29c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:34.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.212+0000 7fa2a0c4e700 1 -- 172.21.15.139:0/427451331 shutdown_connections 2021-11-20T06:00:34.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.212+0000 7fa2a0c4e700 1 -- 172.21.15.139:0/427451331 wait complete. 2021-11-20T06:00:34.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.213+0000 7fa2a0c4e700 1 Processor -- start 2021-11-20T06:00:34.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.213+0000 7fa2a0c4e700 1 -- start start 2021-11-20T06:00:34.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.213+0000 7fa2a0c4e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa29c0f62a0 0x7fa29c1065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:34.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.213+0000 7fa2a0c4e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa29c106aa0 0x7fa29c107990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:34.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.213+0000 7fa2a0c4e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c107000 con 0x7fa29c0f62a0 2021-11-20T06:00:34.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.213+0000 7fa2a0c4e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c107140 con 0x7fa29c106aa0 2021-11-20T06:00:34.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.213+0000 7fa29b7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa29c0f62a0 0x7fa29c1065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:34.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.213+0000 7fa29b7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa29c0f62a0 0x7fa29c1065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56864/0 (socket says 172.21.15.139:56864) 2021-11-20T06:00:34.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.213+0000 7fa29b7fe700 1 -- 172.21.15.139:0/4138942891 learned_addr learned my addr 172.21.15.139:0/4138942891 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:00:34.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.213+0000 7fa29affd700 1 --2- 172.21.15.139:0/4138942891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa29c106aa0 0x7fa29c107990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:34.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.213+0000 7fa29affd700 1 -- 172.21.15.139:0/4138942891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa29c106aa0 msgr2=0x7fa29c107990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-11-20T06:00:34.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.213+0000 7fa29affd700 1 -- 172.21.15.139:0/4138942891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa29c106aa0 msgr2=0x7fa29c107990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-11-20T06:00:34.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.213+0000 7fa29b7fe700 1 -- 172.21.15.139:0/4138942891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa29c106aa0 msgr2=0x7fa29c107990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:34.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.213+0000 7fa29affd700 1 --2- 172.21.15.139:0/4138942891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa29c106aa0 0x7fa29c107990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T06:00:34.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.214+0000 7fa29affd700 1 --2- 172.21.15.139:0/4138942891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa29c106aa0 0x7fa29c107990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T06:00:34.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.214+0000 7fa29b7fe700 1 --2- 172.21.15.139:0/4138942891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa29c106aa0 0x7fa29c107990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:34.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.214+0000 7fa29b7fe700 1 -- 172.21.15.139:0/4138942891 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28c005040 con 0x7fa29c0f62a0 2021-11-20T06:00:34.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.214+0000 7fa29b7fe700 1 --2- 172.21.15.139:0/4138942891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa29c0f62a0 0x7fa29c1065b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fa28c000c00 tx=0x7fa28c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:34.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.214+0000 7fa283fff700 1 -- 172.21.15.139:0/4138942891 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa28c019070 con 0x7fa29c0f62a0 2021-11-20T06:00:34.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.214+0000 7fa283fff700 1 -- 172.21.15.139:0/4138942891 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa28c007c70 con 0x7fa29c0f62a0 2021-11-20T06:00:34.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.214+0000 7fa2a0c4e700 1 -- 172.21.15.139:0/4138942891 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29c107f40 con 0x7fa29c0f62a0 2021-11-20T06:00:34.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.214+0000 7fa283fff700 1 -- 172.21.15.139:0/4138942891 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa28c01d620 con 0x7fa29c0f62a0 2021-11-20T06:00:34.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.214+0000 7fa2a0c4e700 1 -- 172.21.15.139:0/4138942891 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa29c1084a0 con 0x7fa29c0f62a0 2021-11-20T06:00:34.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.215+0000 7fa2a0c4e700 1 -- 172.21.15.139:0/4138942891 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29c059100 con 0x7fa29c0f62a0 2021-11-20T06:00:34.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.215+0000 7fa283fff700 1 -- 172.21.15.139:0/4138942891 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa28c025070 con 0x7fa29c0f62a0 2021-11-20T06:00:34.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.216+0000 7fa283fff700 1 --2- 172.21.15.139:0/4138942891 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa28405c400 0x7fa28405e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:34.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.216+0000 7fa283fff700 1 -- 172.21.15.139:0/4138942891 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa28c0761f0 con 0x7fa29c0f62a0 2021-11-20T06:00:34.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.216+0000 7fa29affd700 1 --2- 172.21.15.139:0/4138942891 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa28405c400 0x7fa28405e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:34.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.216+0000 7fa29affd700 1 --2- 172.21.15.139:0/4138942891 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa28405c400 0x7fa28405e8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fa290006c30 tx=0x7fa290008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:34.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.220+0000 7fa283fff700 1 -- 172.21.15.139:0/4138942891 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa28c049ee0 con 0x7fa29c0f62a0 2021-11-20T06:00:34.365 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.378+0000 7fa2a0c4e700 1 -- 172.21.15.139:0/4138942891 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa29c10cff0 con 0x7fa29c0f62a0 2021-11-20T06:00:34.365 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.379+0000 7fa283fff700 1 -- 172.21.15.139:0/4138942891 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v12) v1 ==== 76+0+227 (secure 0 0 0) 0x7fa28c023030 con 0x7fa29c0f62a0 2021-11-20T06:00:34.366 INFO:teuthology.orchestra.run.smithi139.stdout:# minimal ceph.conf for d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T06:00:34.366 INFO:teuthology.orchestra.run.smithi139.stdout:[global] 2021-11-20T06:00:34.366 INFO:teuthology.orchestra.run.smithi139.stdout: fsid = d819f2cc-49c6-11ec-8c2c-001a4aab830c 2021-11-20T06:00:34.366 INFO:teuthology.orchestra.run.smithi139.stdout: mon_host = [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2021-11-20T06:00:34.368 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.381+0000 7fa2a0c4e700 1 -- 172.21.15.139:0/4138942891 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa28405c400 msgr2=0x7fa28405e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:34.368 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.381+0000 7fa2a0c4e700 1 --2- 172.21.15.139:0/4138942891 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa28405c400 0x7fa28405e8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fa290006c30 tx=0x7fa290008040).stop 2021-11-20T06:00:34.368 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.381+0000 7fa2a0c4e700 1 -- 172.21.15.139:0/4138942891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa29c0f62a0 msgr2=0x7fa29c1065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:34.368 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.381+0000 7fa2a0c4e700 1 --2- 172.21.15.139:0/4138942891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa29c0f62a0 0x7fa29c1065b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fa28c000c00 tx=0x7fa28c00b040).stop 2021-11-20T06:00:34.369 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.381+0000 7fa2a0c4e700 1 -- 172.21.15.139:0/4138942891 shutdown_connections 2021-11-20T06:00:34.370 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.381+0000 7fa2a0c4e700 1 --2- 172.21.15.139:0/4138942891 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa28405c400 0x7fa28405e8c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:34.370 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.381+0000 7fa2a0c4e700 1 --2- 172.21.15.139:0/4138942891 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa29c0f62a0 0x7fa29c1065b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:34.370 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.381+0000 7fa2a0c4e700 1 --2- 172.21.15.139:0/4138942891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa29c106aa0 0x7fa29c107990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:34.371 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.381+0000 7fa2a0c4e700 1 -- 172.21.15.139:0/4138942891 >> 172.21.15.139:0/4138942891 conn(0x7fa29c0f1aa0 msgr2=0x7fa29c0f24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:34.371 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.381+0000 7fa2a0c4e700 1 -- 172.21.15.139:0/4138942891 shutdown_connections 2021-11-20T06:00:34.371 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:34.381+0000 7fa2a0c4e700 1 -- 172.21.15.139:0/4138942891 wait complete. 2021-11-20T06:00:34.388 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:34 smithi139 conmon[33501]: cluster 2021-11-20T06:00:32.179503+0000 mgr.smithi139.dplyxn (mgr.14162) 43 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:34.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:34 smithi178 conmon[36600]: cluster 2021-11-20T06:00:32.179503+0000 mgr.smithi139.dplyxn 2021-11-20T06:00:34.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:34 smithi178 conmon[36600]: (mgr.14162) 43 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:34.708 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2021-11-20T06:00:34.709 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-11-20T06:00:34.710 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.conf 2021-11-20T06:00:34.740 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-11-20T06:00:34.742 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-11-20T06:00:34.805 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2021-11-20T06:00:34.805 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.conf 2021-11-20T06:00:34.833 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2021-11-20T06:00:34.834 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-11-20T06:00:34.900 INFO:tasks.cephadm:Deploying OSDs... 2021-11-20T06:00:34.901 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-11-20T06:00:34.901 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/scratch_devs of=/dev/stdout 2021-11-20T06:00:34.918 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-11-20T06:00:34.919 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-11-20T06:00:34.973 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-11-20T06:00:34.974 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T06:00:34.974 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 115938 Links: 1 2021-11-20T06:00:34.975 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T06:00:34.976 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T06:00:34.976 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2021-11-20 06:00:05.522742404 +0000 2021-11-20T06:00:34.976 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2021-11-20 05:54:08.368845769 +0000 2021-11-20T06:00:34.976 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2021-11-20 05:54:08.368845769 +0000 2021-11-20T06:00:34.977 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2021-11-20T06:00:34.984 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2021-11-20T06:00:34.984 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2021-11-20T06:00:34.984 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000219435 s, 2.3 MB/s 2021-11-20T06:00:34.987 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-11-20T06:00:35.043 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-11-20T06:00:35.043 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T06:00:35.044 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 111448 Links: 1 2021-11-20T06:00:35.044 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T06:00:35.044 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T06:00:35.044 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2021-11-20 06:00:05.522742404 +0000 2021-11-20T06:00:35.045 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2021-11-20 05:54:08.709838558 +0000 2021-11-20T06:00:35.045 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2021-11-20 05:54:08.709838558 +0000 2021-11-20T06:00:35.045 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2021-11-20T06:00:35.053 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2021-11-20T06:00:35.053 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2021-11-20T06:00:35.053 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000223877 s, 2.3 MB/s 2021-11-20T06:00:35.056 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-11-20T06:00:35.111 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-11-20T06:00:35.111 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T06:00:35.112 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 116073 Links: 1 2021-11-20T06:00:35.112 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T06:00:35.112 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T06:00:35.113 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2021-11-20 06:00:05.522742404 +0000 2021-11-20T06:00:35.113 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2021-11-20 05:54:09.044831474 +0000 2021-11-20T06:00:35.113 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2021-11-20 05:54:09.044831474 +0000 2021-11-20T06:00:35.113 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2021-11-20T06:00:35.120 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2021-11-20T06:00:35.121 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2021-11-20T06:00:35.121 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000226049 s, 2.3 MB/s 2021-11-20T06:00:35.125 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-11-20T06:00:35.181 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-11-20T06:00:35.181 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T06:00:35.181 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 117344 Links: 1 2021-11-20T06:00:35.182 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T06:00:35.182 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T06:00:35.182 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2021-11-20 06:00:05.522742404 +0000 2021-11-20T06:00:35.182 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2021-11-20 05:54:09.378824411 +0000 2021-11-20T06:00:35.183 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2021-11-20 05:54:09.378824411 +0000 2021-11-20T06:00:35.183 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2021-11-20T06:00:35.190 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2021-11-20T06:00:35.191 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2021-11-20T06:00:35.191 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000161394 s, 3.2 MB/s 2021-11-20T06:00:35.193 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2021-11-20T06:00:35.194 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2021-11-20T06:00:35.210 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-11-20T06:00:35.211 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-11-20T06:00:35.267 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-11-20T06:00:35.268 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T06:00:35.268 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 113177 Links: 1 2021-11-20T06:00:35.268 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T06:00:35.268 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T06:00:35.269 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2021-11-20 06:00:31.883157177 +0000 2021-11-20T06:00:35.269 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2021-11-20 05:54:08.699817430 +0000 2021-11-20T06:00:35.269 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2021-11-20 05:54:08.699817430 +0000 2021-11-20T06:00:35.269 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2021-11-20T06:00:35.279 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2021-11-20T06:00:35.280 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2021-11-20T06:00:35.280 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 9.9838e-05 s, 5.1 MB/s 2021-11-20T06:00:35.280 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:35 smithi139 conmon[33501]: cluster 2021-11-20T06:00:34.179784+0000 2021-11-20T06:00:35.281 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:35 smithi139 conmon[33501]: mgr.smithi139.dplyxn (mgr.14162) 44 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:35.281 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:35 smithi139 conmon[33501]: audit 2021-11-20T06:00:34.210887+0000 mon.smithi139 (mon.0) 184 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:35.281 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:35 smithi139 conmon[33501]: audit 2021-11-20T06:00:34.211666+0000 mon.smithi139 (mon.0) 185 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:00:35.281 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:35 smithi139 conmon[33501]: audit 2021-11-20T06:00:34.379840+0000 mon.smithi139 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.139:0/4138942891' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:00:35.283 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-11-20T06:00:35.338 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-11-20T06:00:35.339 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T06:00:35.339 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 113256 Links: 1 2021-11-20T06:00:35.339 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T06:00:35.339 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T06:00:35.339 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2021-11-20 06:00:31.883157177 +0000 2021-11-20T06:00:35.340 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2021-11-20 05:54:09.039811607 +0000 2021-11-20T06:00:35.340 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2021-11-20 05:54:09.039811607 +0000 2021-11-20T06:00:35.340 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2021-11-20T06:00:35.348 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2021-11-20T06:00:35.348 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2021-11-20T06:00:35.348 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 8.4295e-05 s, 6.1 MB/s 2021-11-20T06:00:35.351 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-11-20T06:00:35.407 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-11-20T06:00:35.407 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T06:00:35.407 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 112597 Links: 1 2021-11-20T06:00:35.408 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T06:00:35.408 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T06:00:35.408 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2021-11-20 06:00:31.883157177 +0000 2021-11-20T06:00:35.408 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2021-11-20 05:54:09.383805715 +0000 2021-11-20T06:00:35.408 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2021-11-20 05:54:09.383805715 +0000 2021-11-20T06:00:35.409 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2021-11-20T06:00:35.417 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2021-11-20T06:00:35.417 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2021-11-20T06:00:35.418 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 8.1934e-05 s, 6.2 MB/s 2021-11-20T06:00:35.420 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-11-20T06:00:35.477 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-11-20T06:00:35.477 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-20T06:00:35.477 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 118811 Links: 1 2021-11-20T06:00:35.478 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-20T06:00:35.478 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2021-11-20T06:00:35.478 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2021-11-20 06:00:31.883157177 +0000 2021-11-20T06:00:35.478 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2021-11-20 05:54:09.724799875 +0000 2021-11-20T06:00:35.478 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2021-11-20 05:54:09.724799875 +0000 2021-11-20T06:00:35.479 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2021-11-20T06:00:35.484 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:35 smithi178 conmon[36600]: cluster 2021-11-20T06:00:34.179784+0000 mgr.smithi139.dplyxn (mgr.14162) 44 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:35.484 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:35 smithi178 conmon[36600]: audit 2021-11-20T06:00:34.210887+0000 mon.smithi139 2021-11-20T06:00:35.485 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:35 smithi178 conmon[36600]: (mon.0) 184 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:35.485 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:35 smithi178 conmon[36600]: audit 2021-11-20T06:00:34.211666+0000 mon.smithi139 (mon.0) 185 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:00:35.485 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:35 smithi178 conmon[36600]: audit 2021-11-20T06:00:34.379840+0000 mon.smithi139 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.139:0/4138942891' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:00:35.486 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2021-11-20T06:00:35.486 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2021-11-20T06:00:35.487 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 8.4165e-05 s, 6.1 MB/s 2021-11-20T06:00:35.489 INFO:tasks.cephadm:Deploying osd.0 on smithi139 with /dev/vg_nvme/lv_4... 2021-11-20T06:00:35.490 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-11-20T06:00:37.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:37 smithi139 conmon[33501]: cluster 2021-11-20T06:00:36.180000+0000 mgr.smithi139.dplyxn (mgr.14162) 45 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:37.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:37 smithi139 conmon[33501]: audit 2021-11-20T06:00:36.257924+0000 mon.smithi139 (mon.0) 187 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:37.572 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-11-20T06:00:37.572 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-11-20T06:00:37.573 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2021-11-20T06:00:37.573 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2021-11-20T06:00:37.574 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110478 s, 949 MB/s 2021-11-20T06:00:37.574 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2021-11-20T06:00:37.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:37 smithi178 conmon[36600]: cluster 2021-11-20T06:00:36.180000+0000 mgr.smithi139.dplyxn (mgr.14162) 45 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:37.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:37 smithi178 conmon[36600]: audit 2021-11-20T06:00:36.257924+0000 mon.smithi139 (mon.0) 187 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:38.011 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi139:vg_nvme/lv_4 2021-11-20T06:00:39.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:39 smithi139 conmon[33501]: cluster 2021-11-20T06:00:38.180248+0000 mgr.smithi139.dplyxn (mgr.14162) 46 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:39.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:39 smithi178 conmon[36600]: cluster 2021-11-20T06:00:38.180248+0000 mgr.smithi139.dplyxn 2021-11-20T06:00:39.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:39 smithi178 conmon[36600]: (mgr.14162) 46 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:40.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.122+0000 7fd78401f700 1 -- 172.21.15.139:0/3755365086 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd77c0f7980 msgr2=0x7fd77c0f7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:40.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.122+0000 7fd78401f700 1 --2- 172.21.15.139:0/3755365086 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd77c0f7980 0x7fd77c0f7de0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fd76c004660 tx=0x7fd76c009b30).stop 2021-11-20T06:00:40.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.122+0000 7fd78401f700 1 -- 172.21.15.139:0/3755365086 shutdown_connections 2021-11-20T06:00:40.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.122+0000 7fd78401f700 1 --2- 172.21.15.139:0/3755365086 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd77c0f7980 0x7fd77c0f7de0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:40.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.122+0000 7fd78401f700 1 --2- 172.21.15.139:0/3755365086 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd77c0f66e0 0x7fd77c0f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:40.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.122+0000 7fd78401f700 1 -- 172.21.15.139:0/3755365086 >> 172.21.15.139:0/3755365086 conn(0x7fd77c0f1e40 msgr2=0x7fd77c0f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:40.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.122+0000 7fd78401f700 1 -- 172.21.15.139:0/3755365086 shutdown_connections 2021-11-20T06:00:40.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.122+0000 7fd78401f700 1 -- 172.21.15.139:0/3755365086 wait complete. 2021-11-20T06:00:40.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.122+0000 7fd78401f700 1 Processor -- start 2021-11-20T06:00:40.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.122+0000 7fd78401f700 1 -- start start 2021-11-20T06:00:40.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd78401f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd77c0f66e0 0x7fd77c106f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:40.114 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd78401f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd77c0f7980 0x7fd77c107470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:40.114 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd78401f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c107a20 con 0x7fd77c0f7980 2021-11-20T06:00:40.114 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd78401f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c107b60 con 0x7fd77c0f66e0 2021-11-20T06:00:40.115 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd7815ba700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd77c0f7980 0x7fd77c107470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:40.115 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd7815ba700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd77c0f7980 0x7fd77c107470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56876/0 (socket says 172.21.15.139:56876) 2021-11-20T06:00:40.115 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd7815ba700 1 -- 172.21.15.139:0/453023372 learned_addr learned my addr 172.21.15.139:0/453023372 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:00:40.115 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd781dbb700 1 --2- 172.21.15.139:0/453023372 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd77c0f66e0 0x7fd77c106f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:40.116 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd7815ba700 1 -- 172.21.15.139:0/453023372 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd77c0f66e0 msgr2=0x7fd77c106f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:40.116 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd7815ba700 1 --2- 172.21.15.139:0/453023372 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd77c0f66e0 0x7fd77c106f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:40.116 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd7815ba700 1 -- 172.21.15.139:0/453023372 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c005040 con 0x7fd77c0f7980 2021-11-20T06:00:40.116 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd7815ba700 1 --2- 172.21.15.139:0/453023372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd77c0f7980 0x7fd77c107470 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fd76c009f50 tx=0x7fd76c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:40.117 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd7727fc700 1 -- 172.21.15.139:0/453023372 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd76c018070 con 0x7fd77c0f7980 2021-11-20T06:00:40.117 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd7727fc700 1 -- 172.21.15.139:0/453023372 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd76c013410 con 0x7fd77c0f7980 2021-11-20T06:00:40.117 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd78401f700 1 -- 172.21.15.139:0/453023372 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd77c1084a0 con 0x7fd77c0f7980 2021-11-20T06:00:40.117 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd7727fc700 1 -- 172.21.15.139:0/453023372 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd76c01c3c0 con 0x7fd77c0f7980 2021-11-20T06:00:40.118 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.123+0000 7fd78401f700 1 -- 172.21.15.139:0/453023372 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd77c108a00 con 0x7fd77c0f7980 2021-11-20T06:00:40.118 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.125+0000 7fd78401f700 1 -- 172.21.15.139:0/453023372 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77c045ad0 con 0x7fd77c0f7980 2021-11-20T06:00:40.118 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.125+0000 7fd7727fc700 1 -- 172.21.15.139:0/453023372 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd76c01d420 con 0x7fd77c0f7980 2021-11-20T06:00:40.118 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.125+0000 7fd7727fc700 1 --2- 172.21.15.139:0/453023372 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd76805c2b0 0x7fd76805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:00:40.118 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.125+0000 7fd7727fc700 1 -- 172.21.15.139:0/453023372 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd76c076a60 con 0x7fd77c0f7980 2021-11-20T06:00:40.119 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.125+0000 7fd781dbb700 1 --2- 172.21.15.139:0/453023372 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd76805c2b0 0x7fd76805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:00:40.119 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.125+0000 7fd781dbb700 1 --2- 172.21.15.139:0/453023372 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd76805c2b0 0x7fd76805e770 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fd77c0f77e0 tx=0x7fd778009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:00:40.120 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.129+0000 7fd7727fc700 1 -- 172.21.15.139:0/453023372 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd76c04a7b0 con 0x7fd77c0f7980 2021-11-20T06:00:40.310 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:40.324+0000 7fd78401f700 1 -- 172.21.15.139:0/453023372 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd77c108c70 con 0x7fd76805c2b0 2021-11-20T06:00:40.645 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:40 smithi139 conmon[33501]: audit 2021-11-20T06:00:39.474646+0000 mon.smithi139 (mon.0) 188 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:40.645 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:40 smithi139 conmon[33501]: audit 2021-11-20T06:00:39.850053+0000 mon.smithi139 (mon.0) 189 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:40.645 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:40 smithi139 conmon[33501]: cephadm 2021-11-20T06:00:39.851672+0000 mgr.smithi139.dplyxn (mgr.14162) 47 : cephadm [INF] Reconfiguring mon.smithi139 (unknown last config time)... 2021-11-20T06:00:40.646 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:40 smithi139 conmon[33501]: audit 2021-11-20T06:00:39.851877+0000 mon.smithi139 (mon.0) 190 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T06:00:40.646 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:40 smithi139 conmon[33501]: audit 2021-11-20T06:00:39.852439+0000 mon.smithi139 (mon.0) 191 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T06:00:40.646 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:40 smithi139 conmon[33501]: audit 2021-11-20T06:00:39.854271+0000 mon.smithi139 (mon.0) 192 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:00:40.646 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:40 smithi139 conmon[33501]: cephadm 2021-11-20T06:00:39.855074+0000 mgr.smithi139.dplyxn (mgr.14162) 48 : cephadm [INF] Reconfiguring daemon mon.smithi139 on smithi139 2021-11-20T06:00:40.647 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:40 smithi139 conmon[33501]: audit 2021-11-20T06:00:40.326232+0000 mon.smithi139 (mon.0) 193 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T06:00:40.647 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:40 smithi139 conmon[33501]: audit 2021-11-20T 2021-11-20T06:00:40.647 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:40 smithi139 conmon[33501]: 06:00:40.327714+0000 mon.smithi139 (mon.0) 194 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T06:00:40.647 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:40 smithi139 conmon[33501]: audit 2021-11-20T06:00:40.328247+0000 mon.smithi139 (mon.0) 195 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:00:40.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: audit 2021-11-20T06:00:39.474646+0000 mon.smithi139 2021-11-20T06:00:40.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: (mon.0) 188 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:40.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: audit 2021-11-20T06:00:39.850053+0000 mon.smithi139 (mon.0 2021-11-20T06:00:40.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: ) 189 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:40.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: cephadm 2021-11-20T06:00:40.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: 2021-11-20T06:00:39.851672+0000 mgr.smithi139.dplyxn (mgr.14162) 47 : cephadm [INF] 2021-11-20T06:00:40.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: Reconfiguring mon.smithi139 (unknown last config time)... 2021-11-20T06:00:40.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: audit 2021-11-20T06: 2021-11-20T06:00:40.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: 00:39.851877+0000 mon.smithi139 (mon.0) 190 : audit 2021-11-20T06:00:40.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T06:00:40.948 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: audit 2021-11-20T06:00 2021-11-20T06:00:40.948 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: :39.852439+0000 mon.smithi139 (mon.0) 191 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T06:00:40.949 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: 2021-11-20T06:00:40.949 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: audit 2021-11-20T06:00:39.854271 2021-11-20T06:00:40.949 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: +0000 mon.smithi139 (mon.0) 192 : audit [DBG] 2021-11-20T06:00:40.950 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:00:40.950 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: cephadm 2021-11-20T06:00:40.951 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: 2021-11-20T06:00:39.855074+0000 2021-11-20T06:00:40.951 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: mgr.smithi139.dplyxn (mgr.14162) 48 : cephadm [INF] Reconfiguring daemon mon.smithi139 on smithi139 2021-11-20T06:00:40.952 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: audit 2021-11-20T06:00:40.952 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: 2021-11-20T06:00:40.326232+0000 2021-11-20T06:00:40.953 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: mon.smithi139 (mon.0) 193 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T06:00:40.953 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: audit 2021-11-20T06:00:40.954 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: 2021-11-20T06:00:40.327714+0000 mon.smithi139 ( 2021-11-20T06:00:40.954 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: mon.0) 194 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T06:00:40.955 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: audit 2021-11-20T06:00:40.955 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: 2021-11-20T06:00:40.328247+0000 mon.smithi139 (mon 2021-11-20T06:00:40.956 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:40 smithi178 conmon[36600]: .0) 195 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:00:41.851 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: cluster 2021-11-20T06:00:40.180659+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:00:41.851 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: ) 49 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:41.851 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: audit 2021-11-20T06:00:40.325438+0000 mgr.smithi139.dplyxn (mgr.14162) 50 : audit [DBG] from='client.14212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:00:41.852 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: cephadm 2021-11-20T06:00:40.327521+0000 mgr.smithi139.dplyxn (mgr.14162) 51 : cephadm [INF] Applying service osd.None on host smithi139... 2021-11-20T06:00:41.852 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: audit 2021-11-20T06:00:41.853 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: 2021-11-20T06:00:41.297174+0000 mon.smithi139 (mon.0) 196 : audit 2021-11-20T06:00:41.853 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:41.853 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: audit 2021-11 2021-11-20T06:00:41.854 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: -20T06:00:41.299652+0000 2021-11-20T06:00:41.854 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: mon.smithi139 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi139.dplyxn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T06:00:41.855 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: 2021-11-20T06:00:41.855 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: audit 2021-11-20T 2021-11-20T06:00:41.856 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: 06:00:41.300330+0000 mon.smithi139 (mon 2021-11-20T06:00:41.856 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: .0) 198 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T06:00:41.856 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: audit 2021 2021-11-20T06:00:41.857 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: -11-20T06:00: 2021-11-20T06:00:41.857 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: 41.303297+0000 mon.smithi139 (mon.0) 199 : 2021-11-20T06:00:41.859 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:41 smithi139 conmon[33501]: audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:00:42.192 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: cluster 2021-11-20T06:00:40.180659+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:00:42.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: mgr.14162) 49 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:42.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: audit 2021-11-20T06:00:40.325438+0000 2021-11-20T06:00:42.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: mgr.smithi139.dplyxn (mgr.14162) 50 : audit [DBG] from='client.14212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:00:42.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: cephadm 2021-11-20T06:00:40.327521+0000 mgr.smithi139.dplyxn 2021-11-20T06:00:42.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: (mgr.14162) 51 : cephadm [INF] Applying service osd.None on host smithi139... 2021-11-20T06:00:42.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: audit 2021 2021-11-20T06:00:42.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: -11-20T06:00:41.297174+0000 mon.smithi139 (mon.0 2021-11-20T06:00:42.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: ) 196 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:42.195 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: audit 2021-11-20T06:00:42.195 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: 2021-11-20T06:00:41.299652+0000 mon.smithi139 (mon. 2021-11-20T06:00:42.195 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: 0) 197 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi139.dplyxn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T06:00:42.195 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: audit 2021- 2021-11-20T06:00:42.196 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: 11-20T06:00:41.300330+0000 mon.smithi139 (mon.0) 198 : 2021-11-20T06:00:42.196 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T06:00:42.196 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: audit 2021-11-20T06:00:42.196 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: 2021-11-20T06:00:41.303297+0000 mon.smithi139 ( 2021-11-20T06:00:42.196 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:41 smithi178 conmon[36600]: mon.0) 199 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:00:43.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: cephadm 2021-11-20T06:00:41.298982+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:00:43.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: mgr.14162) 52 : cephadm [INF] Reconfiguring mgr.smithi139.dplyxn (unknown last config time)... 2021-11-20T06:00:43.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: cephadm 2021-11-20T06:00:41.303771+0000 2021-11-20T06:00:43.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: mgr.smithi139.dplyxn (mgr.14162) 53 : cephadm [INF] Reconfiguring daemon mgr.smithi139.dplyxn on smithi139 2021-11-20T06:00:43.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: audit 2021-11-20T06:00:42.744246+0000 mon.smithi139 ( 2021-11-20T06:00:43.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: mon.0) 200 : audit [INF] from='client.? 172.21.15.139:0/574820596' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fece5a6-7a49-451c-beb8-f90723d42033"}]: dispatch 2021-11-20T06:00:43.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: audit 2021- 2021-11-20T06:00:43.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: 11-20T06:00:42.748034+0000 mon.smithi139 (mon.0 2021-11-20T06:00:43.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: ) 201 : audit [INF] from='client.? 172.21.15.139:0/574820596' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fece5a6-7a49-451c-beb8-f90723d42033"}]': finished 2021-11-20T06:00:43.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: cluster 2021-11- 2021-11-20T06:00:43.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: 20T06:00:42.748075+0000 mon.smithi139 (mon.0) 2021-11-20T06:00:43.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: 202 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-11-20T06:00:43.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: audit 2021-11-20T06:00:43.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: 2021-11-20T06:00: 2021-11-20T06:00:43.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: 42.748159+0000 mon.smithi139 (mon.0) 203 : audit 2021-11-20T06:00:43.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:43 smithi178 conmon[36600]: [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T06:00:43.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:43 smithi139 conmon[33501]: cephadm 2021-11-20T06:00:41.298982+0000 mgr.smithi139.dplyxn (mgr.14162) 52 : cephadm [INF] Reconfiguring mgr.smithi139.dplyxn (unknown last config time)... 2021-11-20T06:00:43.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:43 smithi139 conmon[33501]: cephadm 2021-11-20T06:00:41.303771+0000 mgr.smithi139.dplyxn (mgr.14162) 53 : cephadm [INF] Reconfiguring daemon mgr.smithi139.dplyxn on smithi139 2021-11-20T06:00:43.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:43 smithi139 conmon[33501]: audit 2021-11-20T06:00:42.744246+0000 mon.smithi139 (mon.0) 200 : audit [INF] from='client.? 172.21.15.139:0/574820596' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8fece5a6-7a49-451c-beb8-f90723d42033"}]: dispatch 2021-11-20T06:00:43.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:43 smithi139 conmon[33501]: audit 2021-11-20T06:00:42.748034+0000 mon.smithi139 (mon.0) 201 : audit [INF] from='client.? 172.21.15.139:0/574820596' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8fece5a6-7a49-451c-beb8-f90723d42033"}]': finished 2021-11-20T06:00:43.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:43 smithi139 conmon[33501]: cluster 2021-11-20T06:00:42.748075+0000 mon.smithi139 (mon.0) 202 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-11-20T06:00:43.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:43 smithi139 conmon[33501]: audit 2021-11-20T06:00:42.748159+0000 mon.smithi139 (mon.0) 203 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T06:00:44.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:44 smithi178 conmon[36600]: cluster 2021-11-20T06:00:42.180861+0000 mgr.smithi139.dplyxn 2021-11-20T06:00:44.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:44 smithi178 conmon[36600]: (mgr.14162) 54 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:44.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:44 smithi178 conmon[36600]: audit 2021-11-20T06:00:43.162619 2021-11-20T06:00:44.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:44 smithi178 conmon[36600]: +0000 mon.smithi139 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.139:0/5372559' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T06:00:44.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:44 smithi139 conmon[33501]: cluster 2021-11-20T06:00:42.180861+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:00:44.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:44 smithi139 conmon[33501]: ) 54 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:44.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:44 smithi139 conmon[33501]: audit 2021-11-20T06:00:43.162619+0000 mon.smithi139 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.139:0/5372559' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T06:00:46.400 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:46 smithi139 conmon[33501]: cluster 2021-11-20T06:00:44.181140+0000 mgr.smithi139.dplyxn 2021-11-20T06:00:46.401 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:46 smithi139 conmon[33501]: (mgr.14162) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:46.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:46 smithi178 conmon[36600]: cluster 2021-11-20T06:00:44.181140+0000 mgr.smithi139.dplyxn 2021-11-20T06:00:46.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:46 smithi178 conmon[36600]: (mgr.14162) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:48.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:48 smithi178 conmon[36600]: cluster 2021-11-20T06:00:46.181530+0000 mgr.smithi139.dplyxn (mgr.14162) 56 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:48.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:48 smithi178 conmon[36600]: 2021-11-20T06:00:48.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:48 smithi178 conmon[36600]: audit 2021-11-20T06:00:47.984507+0000 mon.smithi139 (mon.0) 205 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:48.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:48 smithi139 conmon[33501]: cluster 2021-11-20T06:00:46.181530+0000 mgr.smithi139.dplyxn (mgr.14162) 56 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:48.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:48 smithi139 conmon[33501]: audit 2021-11-20T06:00:47.984507+0000 mon.smithi139 (mon.0) 205 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:49.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:49 smithi178 conmon[36600]: cephadm 2021-11-20T06:00:47.985063+0000 mgr.smithi139.dplyxn (mgr.14162) 57 : cephadm [INF] Reconfiguring alertmanager.smithi139 (dependencies changed)... 2021-11-20T06:00:49.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:49 smithi178 conmon[36600]: cephadm 2021-11-20T06:00:47.987069+0000 mgr.smithi139.dplyxn (mgr.14162) 58 : cephadm [INF] Reconfiguring daemon alertmanager.smithi139 on smithi139 2021-11-20T06:00:49.470 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:49 smithi139 conmon[33501]: cephadm 2021-11-20T06:00:47.985063+0000 mgr.smithi139.dplyxn 2021-11-20T06:00:49.471 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:49 smithi139 conmon[33501]: (mgr.14162) 57 : cephadm [INF] Reconfiguring alertmanager.smithi139 (dependencies changed)... 2021-11-20T06:00:49.471 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:49 smithi139 conmon[33501]: cephadm 2021-11-20T06:00:47.987069+0000 mgr.smithi139.dplyxn (mgr.14162) 58 : cephadm [INF] Reconfiguring daemon alertmanager.smithi139 on smithi139 2021-11-20T06:00:50.384 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:50 smithi139 conmon[33501]: cluster 2021-11-20T06:00:48.181943+0000 mgr.smithi139.dplyxn (mgr.14162) 59 2021-11-20T06:00:50.384 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:50 smithi139 conmon[33501]: : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:50.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:50 smithi178 conmon[36600]: cluster 2021-11-20T06:00:48.181943+0000 mgr.smithi139.dplyxn (mgr.14162) 59 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:51.431 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:51 smithi139 conmon[33501]: cluster 2021-11-20T06:00 2021-11-20T06:00:51.432 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:51 smithi139 conmon[33501]: :50.182337+0000 mgr.smithi139.dplyxn (mgr.14162) 60 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:51.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:51 smithi178 conmon[36600]: cluster 2021-11-20T06:00:50.182337+0000 mgr.smithi139.dplyxn (mgr.14162) 60 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:52.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:52 smithi139 conmon[33501]: audit 2021-11-20T06:00:51.533057+0000 mon.smithi139 (mon.0) 206 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:52.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:52 smithi139 conmon[33501]: cephadm 2021-11-20T06:00:51.533547+0000 mgr.smithi139.dplyxn (mgr.14162) 61 : cephadm [INF] Reconfiguring crash.smithi139 (monmap changed)... 2021-11-20T06:00:52.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:52 smithi139 conmon[33501]: audit 2021-11-20T06:00:51.533766+0000 mon.smithi139 (mon.0) 207 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi139", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T06:00:52.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:52 smithi139 conmon[33501]: audit 2021-11-20T06:00:51.534366+0000 mon.smithi139 (mon.0) 208 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:00:52.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:52 smithi139 conmon[33501]: cephadm 2021-11-20T06:00:51.534786+0000 mgr.smithi139.dplyxn (mgr.14162) 62 : cephadm [INF] Reconfiguring daemon crash.smithi139 on smithi139 2021-11-20T06:00:52.942 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:52 smithi178 conmon[36600]: audit 2021-11-20T06:00:51.533057+0000 mon.smithi139 (mon.0) 206 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:52.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:52 smithi178 conmon[36600]: cephadm 2021-11-20T06:00:51.533547+0000 mgr.smithi139.dplyxn (mgr.14162) 61 : cephadm [INF] Reconfiguring crash.smithi139 (monmap changed)... 2021-11-20T06:00:52.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:52 smithi178 conmon[36600]: audit 2021-11-20T06:00:51 2021-11-20T06:00:52.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:52 smithi178 conmon[36600]: .533766+0000 mon.smithi139 (mon.0) 207 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi139", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T06:00:52.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:52 smithi178 conmon[36600]: audit 2021-11-20T06:00:51 2021-11-20T06:00:52.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:52 smithi178 conmon[36600]: .534366+0000 mon.smithi139 (mon.0) 208 : audit 2021-11-20T06:00:52.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:52 smithi178 conmon[36600]: [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:00:52.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:52 smithi178 conmon[36600]: cephadm 2021-11-20T06:00:51 2021-11-20T06:00:52.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:52 smithi178 conmon[36600]: .534786+0000 mgr.smithi139.dplyxn (mgr.14162) 62 : cephadm [INF] 2021-11-20T06:00:52.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:52 smithi178 conmon[36600]: Reconfiguring daemon crash.smithi139 on smithi139 2021-11-20T06:00:53.770 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:53 smithi139 conmon[33501]: cluster 2021-11-20T06:00:52.182625 2021-11-20T06:00:53.771 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:53 smithi139 conmon[33501]: +0000 mgr.smithi139.dplyxn (mgr.14162) 63 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:53.771 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:53 smithi139 conmon[33501]: audit 2021-11-20T06:00 2021-11-20T06:00:53.771 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:53 smithi139 conmon[33501]: :52.609571+0000 mon.smithi139 (mon.0) 209 : audit [INF] 2021-11-20T06:00:53.771 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:53 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-11-20T06:00:53.771 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:53 smithi139 conmon[33501]: audit 2021 2021-11-20T06:00:53.772 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:53 smithi139 conmon[33501]: -11-20T06:00:52.610160+0000 2021-11-20T06:00:53.773 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:53 smithi139 conmon[33501]: mon.smithi139 (mon.0) 210 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:00:53.773 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:53 smithi139 conmon[33501]: 2021-11-20T06:00:53.773 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:53 smithi139 conmon[33501]: cephadm 2021-11-20T06 2021-11-20T06:00:53.773 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:53 smithi139 conmon[33501]: :00:52.610655+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:00:53.774 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:53 smithi139 conmon[33501]: 64 : cephadm [INF] Deploying daemon osd.0 on smithi139 2021-11-20T06:00:53.774 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:53 smithi139 conmon[33501]: audit 2021-11-20T 2021-11-20T06:00:53.774 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:53 smithi139 conmon[33501]: 06:00:53.485772+0000 mon.smithi139 (mon.0) 2021-11-20T06:00:53.774 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:53 smithi139 conmon[33501]: 211 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:53.942 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:53 smithi178 conmon[36600]: cluster 2021-11-20T06:00:52.182625+0000 mgr.smithi139.dplyxn (mgr.14162) 63 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:53.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:53 smithi178 conmon[36600]: audit 2021-11-20T06:00:52.609571+0000 mon.smithi139 (mon.0) 209 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-11-20T06:00:53.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:53 smithi178 conmon[36600]: audit 2021-11-20 2021-11-20T06:00:53.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:53 smithi178 conmon[36600]: T06:00:52.610160+0000 mon.smithi139 (mon.0) 210 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:00:53.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:53 smithi178 conmon[36600]: cephadm 2021-11 2021-11-20T06:00:53.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:53 smithi178 conmon[36600]: -20T06:00:52.610655+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:00:53.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:53 smithi178 conmon[36600]: mgr.14162) 64 : cephadm [INF] Deploying daemon osd.0 on smithi139 2021-11-20T06:00:53.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:53 smithi178 conmon[36600]: audit 2021- 2021-11-20T06:00:53.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:53 smithi178 conmon[36600]: 11-20T06:00:53.485772+0000 mon.smithi139 2021-11-20T06:00:53.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:53 smithi178 conmon[36600]: (mon.0) 211 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:54.629 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:54 smithi139 conmon[33501]: cephadm 2021-11-20T06:00:53.486269+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:00:54.629 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:54 smithi139 conmon[33501]: 65 : cephadm [INF] Reconfiguring grafana.smithi139 (dependencies changed)... 2021-11-20T06:00:54.629 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:54 smithi139 conmon[33501]: cephadm 2021-11-20T06:00:53.490417+0000 mgr.smithi139.dplyxn (mgr.14162) 66 : cephadm [INF] Reconfiguring daemon grafana.smithi139 on smithi139 2021-11-20T06:00:54.942 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:54 smithi178 conmon[36600]: cephadm 2021-11-20T06:00:53.486269+0000 mgr.smithi139.dplyxn (mgr.14162) 65 : cephadm [INF] Reconfiguring grafana.smithi139 (dependencies changed)... 2021-11-20T06:00:54.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:54 smithi178 conmon[36600]: cephadm 2021-11-20T06:00:53. 2021-11-20T06:00:54.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:54 smithi178 conmon[36600]: 490417+0000 mgr.smithi139.dplyxn (mgr.14162) 66 : cephadm [INF] Reconfiguring daemon grafana.smithi139 on smithi139 2021-11-20T06:00:55.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:55 smithi139 conmon[33501]: cluster 2021-11-20T06:00:54.182932+0000 mgr.smithi139.dplyxn (mgr 2021-11-20T06:00:55.845 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:55 smithi139 conmon[33501]: .14162) 67 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:55.942 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:55 smithi178 conmon[36600]: cluster 2021-11-20T06:00:54.182932+0000 mgr.smithi139.dplyxn 2021-11-20T06:00:55.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:55 smithi178 conmon[36600]: (mgr.14162) 67 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:56.402 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 0 on host 'smithi139' 2021-11-20T06:00:56.403 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:56.413+0000 7fd7727fc700 1 -- 172.21.15.139:0/453023372 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd77c108c70 con 0x7fd76805c2b0 2021-11-20T06:00:56.403 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:56.416+0000 7fd78401f700 1 -- 172.21.15.139:0/453023372 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd76805c2b0 msgr2=0x7fd76805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:56.404 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:56.416+0000 7fd78401f700 1 --2- 172.21.15.139:0/453023372 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd76805c2b0 0x7fd76805e770 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fd77c0f77e0 tx=0x7fd778009040).stop 2021-11-20T06:00:56.404 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:56.416+0000 7fd78401f700 1 -- 172.21.15.139:0/453023372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd77c0f7980 msgr2=0x7fd77c107470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:00:56.404 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:56.416+0000 7fd78401f700 1 --2- 172.21.15.139:0/453023372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd77c0f7980 0x7fd77c107470 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fd76c009f50 tx=0x7fd76c009b30).stop 2021-11-20T06:00:56.405 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:56.417+0000 7fd78401f700 1 -- 172.21.15.139:0/453023372 shutdown_connections 2021-11-20T06:00:56.405 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:56.417+0000 7fd78401f700 1 --2- 172.21.15.139:0/453023372 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd76805c2b0 0x7fd76805e770 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:56.405 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:56.417+0000 7fd78401f700 1 --2- 172.21.15.139:0/453023372 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd77c0f66e0 0x7fd77c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:56.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:56.417+0000 7fd78401f700 1 --2- 172.21.15.139:0/453023372 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd77c0f7980 0x7fd77c107470 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:00:56.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:56.417+0000 7fd78401f700 1 -- 172.21.15.139:0/453023372 >> 172.21.15.139:0/453023372 conn(0x7fd77c0f1e40 msgr2=0x7fd77c0fb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:00:56.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:56.417+0000 7fd78401f700 1 -- 172.21.15.139:0/453023372 shutdown_connections 2021-11-20T06:00:56.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:00:56.417+0000 7fd78401f700 1 -- 172.21.15.139:0/453023372 wait complete. 2021-11-20T06:00:56.769 DEBUG:teuthology.orchestra.run.smithi139:osd.0> sudo journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.0.service 2021-11-20T06:00:56.772 INFO:tasks.cephadm:Deploying osd.1 on smithi139 with /dev/vg_nvme/lv_3... 2021-11-20T06:00:56.772 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-11-20T06:00:56.800 INFO:journalctl@ceph.osd.0.smithi139.stdout:-- Logs begin at Sat 2021-11-20 05:47:58 UTC. -- 2021-11-20T06:00:57.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:57 smithi178 conmon[36600]: cluster 2021-11-20T06:00:56.183238+0000 mgr.smithi139.dplyxn (mgr.14162) 68 2021-11-20T06:00:57.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:57 smithi178 conmon[36600]: : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:57.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:57 smithi178 conmon[36600]: audit 2021-11-20T06:00:56.410126+0000 mon.smithi139 (mon.0) 212 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:57.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:57 smithi178 conmon[36600]: audit 2021-11-20T06:00:56.414264+0000 mon.smithi139 (mon.0) 213 : audit 2021-11-20T06:00:57.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:57 smithi178 conmon[36600]: [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:57.736 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:57 smithi139 conmon[33501]: cluster 2021-11-20T06:00:56.183238+0000 mgr.smithi139.dplyxn (mgr.14162) 68 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:57.737 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:57 smithi139 conmon[33501]: audit 2021-11-20T06:00:56.410126+0000 mon.smithi139 (mon.0) 212 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:57.737 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:57 smithi139 conmon[33501]: audit 2021-11-20T06:00:56.414264+0000 mon.smithi139 (mon.0) 213 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:00:59.200 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-11-20T06:00:59.200 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-11-20T06:00:59.201 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2021-11-20T06:00:59.201 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2021-11-20T06:00:59.201 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011956 s, 877 MB/s 2021-11-20T06:00:59.201 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2021-11-20T06:00:59.599 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:00:59 smithi139 conmon[33501]: cluster 2021-11-20T06:00:58.183505+0000 mgr.smithi139.dplyxn (mgr.14162) 69 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:00:59.600 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi139:vg_nvme/lv_3 2021-11-20T06:00:59.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:59 smithi178 conmon[36600]: cluster 2021-11-20T06:00:58.183505+0000 mgr.smithi139.dplyxn (mgr.14162) 69 2021-11-20T06:00:59.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:00:59 smithi178 conmon[36600]: : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:01:00.591 INFO:journalctl@ceph.osd.0.smithi139.stdout:Nov 20 06:01:00 smithi139 conmon[50551]: debug 2021-11-20T06:01:00.317+0000 7f1f20c7b080 -1 osd.0 0 log_to_monitors {default=true} 2021-11-20T06:01:01.135 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:01 smithi139 conmon[33501]: audit 2021-11-20T06:01:00.071342+0000 mon.smithi139 (mon. 2021-11-20T06:01:01.136 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:01 smithi139 conmon[33501]: 0) 214 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:01.136 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:01 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:00.072627+0000 mgr.smithi139.dplyxn (mgr.14162) 70 : cephadm [INF] Reconfiguring prometheus.smithi139 (dependencies changed)... 2021-11-20T06:01:01.136 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:01 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:00.074633+0000 2021-11-20T06:01:01.136 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:01 smithi139 conmon[33501]: mgr.smithi139.dplyxn (mgr.14162) 71 : cephadm [INF] Reconfiguring daemon prometheus.smithi139 on smithi139 2021-11-20T06:01:01.137 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:01 smithi139 conmon[33501]: audit 2021 2021-11-20T06:01:01.137 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:01 smithi139 conmon[33501]: -11-20T06:01:00.320354+0000 mon.smithi139 2021-11-20T06:01:01.138 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:01 smithi139 conmon[33501]: (mon.0) 215 : audit [INF] from='osd.0 [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-11-20T06:01:01.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:01 smithi178 conmon[36600]: audit 2021-11-20T06:01:00.071342+0000 mon.smithi139 (mon.0) 214 2021-11-20T06:01:01.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:01 smithi178 conmon[36600]: : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:01.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:01 smithi178 conmon[36600]: cephadm 2021-11-20T06:01:00.072627+0000 mgr.smithi139.dplyxn (mgr.14162) 70 : cephadm [INF] Reconfiguring prometheus.smithi139 (dependencies changed)... 2021-11-20T06:01:01.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:01 smithi178 conmon[36600]: cephadm 2021-11-20T06:01:00.074633+0000 mgr.smithi139.dplyxn (mgr.14162) 71 : cephadm 2021-11-20T06:01:01.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:01 smithi178 conmon[36600]: [INF] Reconfiguring daemon prometheus.smithi139 on smithi139 2021-11-20T06:01:01.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:01 smithi178 conmon[36600]: audit 2021-11-20T06:01:00 2021-11-20T06:01:01.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:01 smithi178 conmon[36600]: .320354+0000 mon.smithi139 (mon.0) 215 : audit [INF] 2021-11-20T06:01:01.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:01 smithi178 conmon[36600]: from='osd.0 [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-11-20T06:01:02.192 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:02 smithi178 conmon[36600]: cluster 2021-11-20T06:01:00.183929+0000 mgr.smithi139.dplyxn (mgr.14162) 72 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:01:02.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:02 smithi178 conmon[36600]: audit 2021-11-20T06:01:01.073255+0000 mon.smithi139 (mon 2021-11-20T06:01:02.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:02 smithi178 conmon[36600]: .0) 216 : audit [INF] from='osd.0 [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-11-20T06:01:02.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:02 smithi178 conmon[36600]: cluster 2021-11-20T06:01:01.073304+0000 mon.smithi139 (mon.0) 217 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-11-20T06:01:02.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:02 smithi178 conmon[36600]: audit 2021-11-20T06:01:01.073632+0000 mon.smithi139 (mon.0) 218 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T06:01:02.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:02 smithi178 conmon[36600]: audit 2021-11-20T06:01:01.073902+0000 mon.smithi139 (mon.0) 219 : audit [INF] from='osd.0 [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2021-11-20T06:01:02.249 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.262+0000 7f3501072700 1 -- 172.21.15.139:0/557587138 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34fc0f55c0 msgr2=0x7f34fc0f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:01:02.250 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.262+0000 7f3501072700 1 --2- 172.21.15.139:0/557587138 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34fc0f55c0 0x7f34fc0f59e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f34ec004660 tx=0x7f34ec009b30).stop 2021-11-20T06:01:02.250 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.262+0000 7f3501072700 1 -- 172.21.15.139:0/557587138 shutdown_connections 2021-11-20T06:01:02.250 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.262+0000 7f3501072700 1 --2- 172.21.15.139:0/557587138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34fc0f6860 0x7f34fc0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:01:02.250 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.262+0000 7f3501072700 1 --2- 172.21.15.139:0/557587138 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34fc0f55c0 0x7f34fc0f59e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:02.251 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.262+0000 7f3501072700 1 -- 172.21.15.139:0/557587138 >> 172.21.15.139:0/557587138 conn(0x7f34fc0f0d40 msgr2=0x7f34fc0f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:01:02.251 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.262+0000 7f3501072700 1 -- 172.21.15.139:0/557587138 shutdown_connections 2021-11-20T06:01:02.251 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.262+0000 7f3501072700 1 -- 172.21.15.139:0/557587138 wait complete. 2021-11-20T06:01:02.251 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.262+0000 7f3501072700 1 Processor -- start 2021-11-20T06:01:02.251 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.263+0000 7f3501072700 1 -- start start 2021-11-20T06:01:02.252 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.263+0000 7f3501072700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34fc0f55c0 0x7f34fc105ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:01:02.252 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.263+0000 7f3501072700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34fc0f6860 0x7f34fc1063d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:01:02.252 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.263+0000 7f3501072700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc106a10 con 0x7f34fc0f6860 2021-11-20T06:01:02.252 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.263+0000 7f3501072700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc1072c0 con 0x7f34fc0f55c0 2021-11-20T06:01:02.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.263+0000 7f34fa59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34fc0f6860 0x7f34fc1063d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:01:02.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.263+0000 7f34fa59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34fc0f6860 0x7f34fc1063d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56922/0 (socket says 172.21.15.139:56922) 2021-11-20T06:01:02.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.263+0000 7f34fa59c700 1 -- 172.21.15.139:0/2657354834 learned_addr learned my addr 172.21.15.139:0/2657354834 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:01:02.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.263+0000 7f34fad9d700 1 --2- 172.21.15.139:0/2657354834 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34fc0f55c0 0x7f34fc105ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:01:02.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.263+0000 7f34fa59c700 1 -- 172.21.15.139:0/2657354834 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34fc0f55c0 msgr2=0x7f34fc105ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:01:02.254 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.263+0000 7f34fa59c700 1 --2- 172.21.15.139:0/2657354834 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34fc0f55c0 0x7f34fc105ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:02.254 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.263+0000 7f34fa59c700 1 -- 172.21.15.139:0/2657354834 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ec005040 con 0x7f34fc0f6860 2021-11-20T06:01:02.254 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.263+0000 7f34fa59c700 1 --2- 172.21.15.139:0/2657354834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34fc0f6860 0x7f34fc1063d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f34f000b410 tx=0x7f34f0006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:01:02.254 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.263+0000 7f34e37fe700 1 -- 172.21.15.139:0/2657354834 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34f00125f0 con 0x7f34fc0f6860 2021-11-20T06:01:02.255 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.264+0000 7f34e37fe700 1 -- 172.21.15.139:0/2657354834 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34f000be30 con 0x7f34fc0f6860 2021-11-20T06:01:02.255 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.264+0000 7f34e37fe700 1 -- 172.21.15.139:0/2657354834 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34f0002a20 con 0x7f34fc0f6860 2021-11-20T06:01:02.256 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.264+0000 7f3501072700 1 -- 172.21.15.139:0/2657354834 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34fc1075a0 con 0x7f34fc0f6860 2021-11-20T06:01:02.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.264+0000 7f3501072700 1 -- 172.21.15.139:0/2657354834 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34fc107b10 con 0x7f34fc0f6860 2021-11-20T06:01:02.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.265+0000 7f3501072700 1 -- 172.21.15.139:0/2657354834 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34fc0f49b0 con 0x7f34fc0f6860 2021-11-20T06:01:02.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.265+0000 7f34e37fe700 1 -- 172.21.15.139:0/2657354834 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f34f000f030 con 0x7f34fc0f6860 2021-11-20T06:01:02.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.265+0000 7f34e37fe700 1 --2- 172.21.15.139:0/2657354834 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f34e405c2a0 0x7f34e405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:01:02.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.265+0000 7f34e37fe700 1 -- 172.21.15.139:0/2657354834 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1602+0+0 (secure 0 0 0) 0x7f34f0075420 con 0x7f34fc0f6860 2021-11-20T06:01:02.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.265+0000 7f34fad9d700 1 --2- 172.21.15.139:0/2657354834 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f34e405c2a0 0x7f34e405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:01:02.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.266+0000 7f34fad9d700 1 --2- 172.21.15.139:0/2657354834 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f34e405c2a0 0x7f34e405e760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f34ec004e30 tx=0x7f34ec009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:01:02.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.270+0000 7f34e37fe700 1 -- 172.21.15.139:0/2657354834 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34f0048fa0 con 0x7f34fc0f6860 2021-11-20T06:01:02.438 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:02.451+0000 7f3501072700 1 -- 172.21.15.139:0/2657354834 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f34fc05f620 con 0x7f34e405c2a0 2021-11-20T06:01:02.439 INFO:journalctl@ceph.osd.0.smithi139.stdout:Nov 20 06:01:02 smithi139 conmon[50551]: debug 2021-11-20T06:01:02.079+0000 7f1f171ec700 -1 osd.0 0 waiting for initial osdmap 2021-11-20T06:01:02.439 INFO:journalctl@ceph.osd.0.smithi139.stdout:Nov 20 06:01:02 smithi139 conmon[50551]: debug 2021-11-20T06:01:02.084+0000 7f1f11964700 -1 osd.0 8 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T06:01:02.439 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:02 smithi139 conmon[33501]: cluster 2021-11-20T06:01:00.183929+0000 2021-11-20T06:01:02.440 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:02 smithi139 conmon[33501]: mgr.smithi139.dplyxn (mgr.14162) 72 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:01:02.440 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:02 smithi139 conmon[33501]: audit 2021-11-20T06:01:01.073255+0000 mon.smithi139 (mon.0) 216 : audit [INF] from='osd.0 [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-11-20T06:01:02.440 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:02 smithi139 conmon[33501]: cluster 2021-11-20T06:01:01.073304+0000 mon.smithi139 (mon.0) 217 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-11-20T06:01:02.441 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:02 smithi139 conmon[33501]: audit 2021-11-20T06:01:01.073632+0000 mon.smithi139 (mon.0) 218 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T06:01:02.441 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:02 smithi139 conmon[33501]: audit 2021-11-20T06:01:01.073902+0000 mon.smithi139 (mon.0) 219 : audit [INF] from='osd.0 [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2021-11-20T06:01:03.336 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:03 smithi139 conmon[33501]: audit 2021-11-20T06:01:02.076615+0000 mon.smithi139 2021-11-20T06:01:03.336 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:03 smithi139 conmon[33501]: (mon.0) 220 : audit [INF] from='osd.0 [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2021-11-20T06:01:03.336 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:03 smithi139 conmon[33501]: cluster 2021-11-20T06:01:02.076683+0000 mon.smithi139 (mon.0) 221 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2021-11-20T06:01:03.337 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:03 smithi139 conmon[33501]: audit 2021-11-20T06:01:02.077234+0000 mon.smithi139 (mon.0) 222 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T06:01:03.337 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:03 smithi139 conmon[33501]: audit 2021-11-20T06:01:02.081998+0000 mon.smithi139 (mon.0) 223 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T06:01:03.337 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:03 smithi139 conmon[33501]: audit 2021-11-20T06: 2021-11-20T06:01:03.337 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:03 smithi139 conmon[33501]: 01:02.454034+0000 mon.smithi139 (mon.0) 224 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T06:01:03.338 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:03 smithi139 conmon[33501]: audit 2021-11-20T06:01 2021-11-20T06:01:03.338 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:03 smithi139 conmon[33501]: :02.455303+0000 mon.smithi139 (mon.0) 225 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T06:01:03.338 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:03 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:01:03.338 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:03 smithi139 conmon[33501]: 20T06:01:02.455811+0000 mon.smithi139 (mon 2021-11-20T06:01:03.338 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:03 smithi139 conmon[33501]: .0) 226 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:03.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:03 smithi178 conmon[36600]: audit 2021-11-20T06:01:02.076615+0000 mon.smithi139 (mon.0) 220 : audit [INF] from='osd.0 [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2021-11-20T06:01:03.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:03 smithi178 conmon[36600]: cluster 2021- 2021-11-20T06:01:03.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:03 smithi178 conmon[36600]: 11-20T06:01:02.076683+0000 mon.smithi139 (mon.0) 221 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2021-11-20T06:01:03.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:03 smithi178 conmon[36600]: audit 2021-11-20T06:01:02.077234+0000 mon.smithi139 (mon.0) 222 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T06:01:03.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:03 smithi178 conmon[36600]: audit 2021-11-20T06:01:02.081998+0000 mon.smithi139 ( 2021-11-20T06:01:03.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:03 smithi178 conmon[36600]: mon.0) 223 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T06:01:03.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:03 smithi178 conmon[36600]: audit 2021-11-20T 2021-11-20T06:01:03.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:03 smithi178 conmon[36600]: 06:01:02.454034+0000 mon.smithi139 (mon.0 2021-11-20T06:01:03.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:03 smithi178 conmon[36600]: ) 224 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T06:01:03.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:03 smithi178 conmon[36600]: audit 2021-11-20T06 2021-11-20T06:01:03.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:03 smithi178 conmon[36600]: :01:02.455303+0000 mon.smithi139 (mon.0) 225 : audit 2021-11-20T06:01:03.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:03 smithi178 conmon[36600]: [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T06:01:03.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:03 smithi178 conmon[36600]: audit 2021-11-20T 2021-11-20T06:01:03.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:03 smithi178 conmon[36600]: 06:01:02.455811+0000 mon.smithi139 (mon.0 2021-11-20T06:01:03.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:03 smithi178 conmon[36600]: ) 226 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:04.335 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: cluster 2021-11-20T06:01:01.290355+0000 osd.0 (osd.0 2021-11-20T06:01:04.335 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: ) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T06:01:04.336 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: cluster 2021-11-20T06:01:01.290564+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T06:01:04.336 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: cluster 2021-11- 2021-11-20T06:01:04.336 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: 20T06:01:02.184148+0000 mgr.smithi139.dplyxn (mgr.14162) 73 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:01:04.337 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: audit 2021-11-20 2021-11-20T06:01:04.338 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: T06:01:02.453126+0000 mgr.smithi139.dplyxn (mgr.14162) 74 : audit [DBG] from='client.14230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:01:04.338 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: cephadm 2021-11-20T06:01 2021-11-20T06:01:04.338 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: :02.455115+0000 mgr.smithi139.dplyxn (mgr.14162) 75 : cephadm [INF] Applying service osd.None on host smithi139... 2021-11-20T06:01:04.339 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: 2021-11-20T06:01:04.339 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: audit 2021-11 2021-11-20T06:01:04.339 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: -20T06:01:03.079749+0000 mon.smithi139 (mon.0) 227 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T06:01:04.340 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: cluster 2021-11-20T06 2021-11-20T06:01:04.340 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: :01:03.085812+0000 mon.smithi139 (mon.0) 228 : cluster [INF] 2021-11-20T06:01:04.340 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: osd.0 [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595] boot 2021-11-20T06:01:04.341 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: cluster 2021-11-20T06:01:04.342 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: 2021-11-20T06:01:03.085834+0000 mon.smithi139 (mon.0) 229 : cluster [DBG] 2021-11-20T06:01:04.342 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: osdmap e9: 1 total, 1 up, 1 in 2021-11-20T06:01:04.343 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: audit 2021-11-20T06:01:04.343 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: 2021-11-20T06:01:03.086062+0000 mon.smithi139 (mon.0) 230 : audit 2021-11-20T06:01:04.343 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T06:01:04.344 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: audit 2021-11-20T06:01:04.344 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: 2021-11-20T06:01:03.910564+0000 mon.smithi139 (mon.0) 231 : audit [INF] 2021-11-20T06:01:04.344 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:04.345 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: audit 2021- 2021-11-20T06:01:04.345 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: 11-20T06:01:03.911318+0000 mon.smithi139 (mon.0) 232 : audit [INF] 2021-11-20T06:01:04.346 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi178", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T06:01:04.346 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: audit 2021-11-20T06:01:04.346 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: 2021-11-20T06:01:03.911969+0000 mon.smithi139 (mon.0) 233 : audit [DBG] 2021-11-20T06:01:04.346 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:04 smithi178 conmon[36600]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:04.373 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:04 smithi139 conmon[33501]: cluster 2021-11-20T06:01:01.290355+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T06:01:04.373 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:04 smithi139 conmon[33501]: cluster 2021-11-20T06 2021-11-20T06:01:04.374 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:04 smithi139 conmon[33501]: :01:01.290564+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T06:01:04.374 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:04 smithi139 conmon[33501]: cluster 2021-11-20T06:01:02.184148+0000 mgr.smithi139.dplyxn (mgr.14162) 73 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-20T06:01:04.374 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:04 smithi139 conmon[33501]: audit 2021-11-20T06:01:02.453126+0000 mgr.smithi139.dplyxn (mgr.14162) 74 : audit [DBG] from='client.14230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:01:04.375 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:04 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:02.455115+0000 mgr.smithi139.dplyxn (mgr.14162) 75 : cephadm [INF] Applying service osd.None on host smithi139... 2021-11-20T06:01:04.375 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:04 smithi139 conmon[33501]: audit 2021-11-20T06:01:03.079749+0000 mon.smithi139 (mon.0) 227 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T06:01:04.375 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:04 smithi139 conmon[33501]: cluster 2021-11-20T06:01:03.085812+0000 mon.smithi139 (mon.0) 228 : cluster [INF] osd.0 [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595] boot 2021-11-20T06:01:04.375 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:04 smithi139 conmon[33501]: cluster 2021-11-20T06:01:03.085834+0000 mon.smithi139 (mon.0) 229 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-11-20T06:01:04.376 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:04 smithi139 conmon[33501]: audit 2021-11-20T06:01:03.086062+0000 mon.smithi139 (mon.0) 230 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-20T06:01:04.376 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:04 smithi139 conmon[33501]: audit 2021-11-20T06:01:03.910564+0000 mon.smithi139 (mon.0) 231 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:04.376 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:04 smithi139 conmon[33501]: audit 2021-11-20T06:01:03.911318+0000 mon.smithi139 (mon.0) 232 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi178", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-20T06:01:04.377 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:04 smithi139 conmon[33501]: audit 2021-11-20T06:01:03.911969+0000 mon.smithi139 (mon.0) 233 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:05.192 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:05 smithi178 conmon[36600]: cephadm 2021-11-20T06:01:03.911049+0000 mgr.smithi139.dplyxn (mgr.14162) 76 : cephadm [INF] Reconfiguring crash.smithi178 (monmap changed)... 2021-11-20T06:01:05.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:05 smithi178 conmon[36600]: 2021-11-20T06:01:05.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:05 smithi178 conmon[36600]: cephadm 2021-11-20T06:01:03.912437+0000 mgr.smithi139.dplyxn (mgr.14162) 77 : cephadm [INF] Reconfiguring daemon crash.smithi178 on smithi178 2021-11-20T06:01:05.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:05 smithi178 conmon[36600]: cluster 2021-11-20T06:01:04.151263+0000 mon.smithi139 (mon.0) 234 : cluster [DBG] osdmap e10: 1 total, 1 up, 1 in 2021-11-20T06:01:05.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:05 smithi178 conmon[36600]: audit 2021-11-20T06:01:04.993087+0000 mon.smithi139 (mon.0) 235 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:05.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:05 smithi178 conmon[36600]: audit 2021-11-20T06:01:04.993856+0000 mon.smithi139 (mon.0) 236 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi178.zxxdkz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T06:01:05.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:05 smithi178 conmon[36600]: audit 2021-11-20T06:01:04.994582+0000 mon.smithi139 (mon.0) 237 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T06:01:05.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:05 smithi178 conmon[36600]: audit 2021-11-20T06:01:04.995155+0000 mon.smithi139 (mon.0) 238 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:05.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:05 smithi178 conmon[36600]: audit 2021-11-20T06:01:05.081342+0000 mon.smithi139 (mon.0) 239 : audit [INF] from='client.? 172.21.15.139:0/3110033071' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce18ef72-7c6c-43f7-8a0c-6904f7c9e606"}]: dispatch 2021-11-20T06:01:05.195 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:05 smithi178 conmon[36600]: audit 2021-11-20T06:01:05.089274+0000 mon.smithi139 (mon.0) 240 : audit [INF] from='client.? 172.21.15.139:0/3110033071' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce18ef72-7c6c-43f7-8a0c-6904f7c9e606"}]': finished 2021-11-20T06:01:05.195 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:05 smithi178 conmon[36600]: cluster 2021-11-20T06:01:05.089317+0000 mon.smithi139 (mon.0) 241 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-11-20T06:01:05.195 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:05 smithi178 conmon[36600]: audit 2021-11-20T06:01:05.195 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:05 smithi178 conmon[36600]: 2021-11-20T06:01:05.089387+0000 mon.smithi139 (mon.0) 242 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T06:01:05.483 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:03.911049+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:01:05.483 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: 14162) 76 : cephadm [INF] Reconfiguring crash.smithi178 (monmap changed)... 2021-11-20T06:01:05.484 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:03.912437+0000 mgr.smithi139.dplyxn (mgr.14162) 77 : cephadm [INF] Reconfiguring daemon crash.smithi178 on smithi178 2021-11-20T06:01:05.484 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: cluster 2021-11-20T06:01:04.151263+0000 mon.smithi139 (mon.0) 234 : cluster [DBG] osdmap e10: 1 total, 1 up, 1 in 2021-11-20T06:01:05.484 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: audit 2021-11-20T06:01:04.993087+0000 mon.smithi139 (mon.0) 235 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:05.485 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: audit 2021-11-20T06:01:04.993856+0000 mon.smithi139 (mon.0) 236 : audit 2021-11-20T06:01:05.485 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi178.zxxdkz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-20T06:01:05.485 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: audit 2021-11-20T06:01:04.994582+0000 mon.smithi139 (mon.0) 237 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-20T06:01:05.486 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: audit 2021-11-20T06:01:04.995155 2021-11-20T06:01:05.486 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 238 : audit [DBG] 2021-11-20T06:01:05.486 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:05.487 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: audit 2021 2021-11-20T06:01:05.487 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: -11-20T06:01:05.081342+0000 mon.smithi139 (mon.0) 2021-11-20T06:01:05.487 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: 239 : audit [INF] from='client.? 172.21.15.139:0/3110033071' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce18ef72-7c6c-43f7-8a0c-6904f7c9e606"}]: dispatch 2021-11-20T06:01:05.487 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: audit 2021-11-20T06:01 2021-11-20T06:01:05.488 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: :05.089274+0000 mon.smithi139 (mon.0) 240 : 2021-11-20T06:01:05.488 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: audit [INF] from='client.? 172.21.15.139:0/3110033071' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce18ef72-7c6c-43f7-8a0c-6904f7c9e606"}]': finished 2021-11-20T06:01:05.488 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: cluster 2021-11-20T06:01:05.489 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: 2021-11-20T06:01:05.089317+0000 mon.smithi139 ( 2021-11-20T06:01:05.489 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: mon.0) 241 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-11-20T06:01:05.489 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: audit 2021-11-20 2021-11-20T06:01:05.489 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: T06:01:05.089387+0000 mon.smithi139 (mon 2021-11-20T06:01:05.490 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:05 smithi139 conmon[33501]: .0) 242 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T06:01:06.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:06 smithi178 conmon[36600]: cluster 2021-11-20T06:01:04.184447+0000 mgr.smithi139.dplyxn (mgr.14162) 78 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:06.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:06 smithi178 conmon[36600]: cephadm 2021-11-20T06:01:04.993585+0000 mgr.smithi139.dplyxn (mgr.14162) 79 : cephadm [INF] Reconfiguring mgr.smithi178.zxxdkz (monmap changed)... 2021-11-20T06:01:06.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:06 smithi178 conmon[36600]: cephadm 2021-11-20T06:01:04.995628+0000 mgr.smithi139.dplyxn (mgr.14162) 80 : cephadm [INF] Reconfiguring daemon mgr.smithi178.zxxdkz on smithi178 2021-11-20T06:01:06.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:06 smithi178 conmon[36600]: audit 2021-11-20T06:01:05.497680+0000 mon.smithi139 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.139:0/3802607786' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T06:01:06.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:06 smithi139 conmon[33501]: cluster 2021-11-20T06:01:04.184447+0000 mgr.smithi139.dplyxn (mgr.14162) 78 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:06.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:06 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:04.993585+0000 mgr.smithi139.dplyxn (mgr.14162) 79 : cephadm [INF] Reconfiguring mgr.smithi178.zxxdkz (monmap changed)... 2021-11-20T06:01:06.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:06 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:04.995628+0000 mgr.smithi139.dplyxn (mgr.14162) 80 : cephadm [INF] Reconfiguring daemon mgr.smithi178.zxxdkz on smithi178 2021-11-20T06:01:06.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:06 smithi139 conmon[33501]: audit 2021-11-20T06:01:05.497680+0000 mon.smithi139 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.139:0/3802607786' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T06:01:07.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:07 smithi178 conmon[36600]: audit 2021-11-20T06:01:06.165816+0000 mon.smithi139 (mon.0) 244 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:07.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:07 smithi178 conmon[36600]: cephadm 2021-11-20T06:01:06.166320+0000 mgr.smithi139.dplyxn (mgr.14162) 81 : cephadm [INF] Reconfiguring mon.smithi178 (monmap changed)... 2021-11-20T06:01:07.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:07 smithi178 conmon[36600]: audit 2021-11-20T06:01:06.166574+0000 mon.smithi139 (mon.0) 245 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T06:01:07.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:07 smithi178 conmon[36600]: audit 2021-11-20T06:01:06.167185+0000 mon.smithi139 (mon.0) 246 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T06:01:07.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:07 smithi178 conmon[36600]: audit 2021-11-20T06:01:06.167742+0000 mon.smithi139 (mon.0) 247 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:07.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:07 smithi178 conmon[36600]: cephadm 2021-11-20T06:01:06.168214+0000 mgr.smithi139.dplyxn (mgr.14162) 82 2021-11-20T06:01:07.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:07 smithi178 conmon[36600]: : cephadm [INF] Reconfiguring daemon mon.smithi178 on smithi178 2021-11-20T06:01:07.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:07 smithi178 conmon[36600]: cluster 2021-11-20T06 2021-11-20T06:01:07.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:07 smithi178 conmon[36600]: :01:06.184785+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:01:07.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:07 smithi178 conmon[36600]: 14162) 83 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:07.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:07 smithi139 conmon[33501]: audit 2021-11-20T06:01:06.165816+0000 mon.smithi139 (mon.0) 244 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:07.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:07 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:06.166320+0000 mgr.smithi139.dplyxn (mgr.14162) 81 : cephadm [INF] Reconfiguring mon.smithi178 (monmap changed)... 2021-11-20T06:01:07.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:07 smithi139 conmon[33501]: audit 2021-11-20T06:01:06.166574+0000 mon.smithi139 (mon.0) 245 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-20T06:01:07.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:07 smithi139 conmon[33501]: audit 2021-11-20T06:01:07.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:07 smithi139 conmon[33501]: 2021-11-20T06:01:06.167185+0000 mon.smithi139 (mon.0) 246 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-20T06:01:07.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:07 smithi139 conmon[33501]: audit 2021-11-20T06:01:06.167742+0000 2021-11-20T06:01:07.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:07 smithi139 conmon[33501]: mon.smithi139 (mon.0) 247 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:07.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:07 smithi139 conmon[33501]: 2021-11-20T06:01:07.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:07 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:07.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:07 smithi139 conmon[33501]: 2021-11-20T06:01:06.168214+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:01:07.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:07 smithi139 conmon[33501]: 14162) 82 : cephadm [INF] Reconfiguring daemon mon.smithi178 on smithi178 2021-11-20T06:01:07.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:07 smithi139 conmon[33501]: cluster 2021-11-20T 2021-11-20T06:01:07.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:07 smithi139 conmon[33501]: 06:01:06.184785+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:01:07.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:07 smithi139 conmon[33501]: ) 83 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:08.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:08 smithi139 conmon[33501]: audit 2021-11-20T06:01:07.236003+0000 mon.smithi139 (mon.0) 248 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:08.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:08 smithi139 conmon[33501]: audit 2021-11-20T06:01:07.239060+0000 mon.smithi139 2021-11-20T06:01:08.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:08 smithi139 conmon[33501]: (mon.0) 249 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T06:01:08.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:08 smithi139 conmon[33501]: audit 2021-11-20T06:01:07.239460+0000 mgr.smithi139.dplyxn (mgr.14162) 84 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T06:01:08.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:08 smithi178 conmon[36600]: audit 2021-11-20T06:01:07.236003+0000 mon.smithi139 (mon.0) 248 : audit [INF] 2021-11-20T06:01:08.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:08 smithi178 conmon[36600]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:08.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:08 smithi178 conmon[36600]: audit 2021-11-20T06:01:07.239060+0000 mon.smithi139 (mon.0) 249 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T06:01:08.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:08 smithi178 conmon[36600]: audit 2021-11-20T06:01:07.239460+0000 mgr.smithi139.dplyxn (mgr.14162) 84 : 2021-11-20T06:01:08.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:08 smithi178 conmon[36600]: audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-20T06:01:09.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:09 smithi139 conmon[33501]: cluster 2021-11-20T06:01:08.185202+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:01:09.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:09 smithi139 conmon[33501]: 14162) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:09.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:09 smithi178 conmon[36600]: cluster 2021-11-20T06:01:08.185202+0000 mgr.smithi139.dplyxn 2021-11-20T06:01:09.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:09 smithi178 conmon[36600]: (mgr.14162) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:11.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:11 smithi139 conmon[33501]: cluster 2021-11-20T06:01:10.185613+0000 mgr.smithi139.dplyxn (mgr 2021-11-20T06:01:11.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:11 smithi139 conmon[33501]: .14162) 86 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:11.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:11 smithi178 conmon[36600]: cluster 2021-11-20T06:01:10.185613+0000 mgr.smithi139.dplyxn (mgr.14162) 86 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:12.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:12 smithi178 conmon[36600]: audit 2021-11-20T06:01:11.370916+0000 mon.smithi139 (mon.0) 250 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-11-20T06:01:12.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:12 smithi178 conmon[36600]: audit 2021-11-20T06:01:11.371443 2021-11-20T06:01:12.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:12 smithi178 conmon[36600]: +0000 mon.smithi139 (mon.0) 251 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:12.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:12 smithi178 conmon[36600]: cephadm 2021-11-20T06:01:11.371871+0000 mgr.smithi139.dplyxn (mgr.14162) 87 : cephadm [INF] Deploying daemon osd.1 on smithi139 2021-11-20T06:01:12.499 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:12 smithi139 conmon[33501]: audit 2021-11-20T06:01:11.370916+0000 mon.smithi139 (mon.0) 250 : audit 2021-11-20T06:01:12.499 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:12 smithi139 conmon[33501]: [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-11-20T06:01:12.499 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:12 smithi139 conmon[33501]: audit 2021-11-20T06:01:11.371443+0000 mon.smithi139 (mon.0) 251 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:12.500 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:12 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:11.371871+0000 mgr.smithi139.dplyxn (mgr.14162) 87 : cephadm [INF] Deploying daemon osd.1 on smithi139 2021-11-20T06:01:14.413 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:14 smithi139 conmon[33501]: cluster 2021-11-20T06:01:12.185883+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:01:14.413 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:14 smithi139 conmon[33501]: 88 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:14.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:14 smithi178 conmon[36600]: cluster 2021-11-20T06:01:12.185883+0000 mgr.smithi139.dplyxn (mgr.14162) 88 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:14.466 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 1 on host 'smithi139' 2021-11-20T06:01:14.468 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:14.478+0000 7f34e37fe700 1 -- 172.21.15.139:0/2657354834 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f34fc05f620 con 0x7f34e405c2a0 2021-11-20T06:01:14.468 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:14.480+0000 7f3501072700 1 -- 172.21.15.139:0/2657354834 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f34e405c2a0 msgr2=0x7f34e405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:01:14.469 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:14.480+0000 7f3501072700 1 --2- 172.21.15.139:0/2657354834 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f34e405c2a0 0x7f34e405e760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f34ec004e30 tx=0x7f34ec009b30).stop 2021-11-20T06:01:14.469 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:14.480+0000 7f3501072700 1 -- 172.21.15.139:0/2657354834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34fc0f6860 msgr2=0x7f34fc1063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:01:14.469 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:14.480+0000 7f3501072700 1 --2- 172.21.15.139:0/2657354834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34fc0f6860 0x7f34fc1063d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f34f000b410 tx=0x7f34f0006ef0).stop 2021-11-20T06:01:14.470 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:14.481+0000 7f3501072700 1 -- 172.21.15.139:0/2657354834 shutdown_connections 2021-11-20T06:01:14.470 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:14.481+0000 7f3501072700 1 --2- 172.21.15.139:0/2657354834 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f34e405c2a0 0x7f34e405e760 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:14.470 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:14.481+0000 7f3501072700 1 --2- 172.21.15.139:0/2657354834 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34fc0f55c0 0x7f34fc105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:14.470 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:14.481+0000 7f3501072700 1 --2- 172.21.15.139:0/2657354834 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f34fc0f6860 0x7f34fc1063d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:14.471 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:14.481+0000 7f3501072700 1 -- 172.21.15.139:0/2657354834 >> 172.21.15.139:0/2657354834 conn(0x7f34fc0f0d40 msgr2=0x7f34fc0fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:01:14.471 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:14.481+0000 7f3501072700 1 -- 172.21.15.139:0/2657354834 shutdown_connections 2021-11-20T06:01:14.471 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:14.481+0000 7f3501072700 1 -- 172.21.15.139:0/2657354834 wait complete. 2021-11-20T06:01:14.766 DEBUG:teuthology.orchestra.run.smithi139:osd.1> sudo journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.1.service 2021-11-20T06:01:14.769 INFO:tasks.cephadm:Deploying osd.2 on smithi139 with /dev/vg_nvme/lv_2... 2021-11-20T06:01:14.770 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-11-20T06:01:14.797 INFO:journalctl@ceph.osd.1.smithi139.stdout:-- Logs begin at Sat 2021-11-20 05:47:58 UTC. -- 2021-11-20T06:01:15.028 INFO:journalctl@ceph.osd.1.smithi139.stdout:Nov 20 06:01:14 smithi139 conmon[55514]: debug 2021-11-20T06:01:14.955+0000 7f4407443080 -1 Falling back to public interface 2021-11-20T06:01:15.543 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: cluster 2021-11-20T06:01:14.186224+0000 mgr.smithi139.dplyxn (mgr.14162) 89 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:15.543 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021-11-20T06:01:14.474846+0000 mon.smithi139 (mon.0) 252 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:15.543 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021-11-20T06:01:14.478333+0000 mon.smithi139 (mon.0) 253 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:15.544 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021-11-20T06:01:14.492348+0000 mon.smithi139 (mon.0) 254 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.139:9093"}]: dispatch 2021-11-20T06:01:15.544 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021-11-20T06:01:14.492691+0000 mgr.smithi139.dplyxn (mgr.14162) 90 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.139:9093"}]: dispatch 2021-11-20T06:01:15.544 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021-11-20T06:01:14.497360+0000 mon.smithi139 (mon.0) 255 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:15.544 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021-11-20T06:01:14.500586+0000 mon.smithi139 (mon.0) 256 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T06:01:15.545 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021-11-20T06:01:14 2021-11-20T06:01:15.545 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: .500900+0000 mgr.smithi139.dplyxn (mgr.14162) 91 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T06:01:15.545 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021-11-20T06:01:14.505452+0000 mon.smithi139 (mon.0) 257 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.139:3000"}]: dispatch 2021-11-20T06:01:15.545 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021-11-20T06:01:14.505776+0000 mgr.smithi139.dplyxn (mgr.14162) 92 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.139:3000"}]: dispatch 2021-11-20T06:01:15.545 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021-11-20T06:01:14.510074+0000 mon.smithi139 (mon.0) 258 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:15.546 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021-11-20T06:01:14.513374+0000 mon.smithi139 (mon.0) 259 : audit 2021-11-20T06:01:15.546 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T06:01:15.546 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021-11-20T06:01:14.513780+0000 mgr.smithi139.dplyxn (mgr.14162) 93 2021-11-20T06:01:15.546 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T06:01:15.547 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021-11-20T06:01:15.547 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: 2021-11-20T06:01:14.521006+0000 mon.smithi139 ( 2021-11-20T06:01:15.547 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: mon.0) 260 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.139:9095"}]: dispatch 2021-11-20T06:01:15.547 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021- 2021-11-20T06:01:15.548 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: 11-20T06:01:14.521363+0000 2021-11-20T06:01:15.548 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: mgr.smithi139.dplyxn (mgr.14162) 94 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.139:9095"}]: dispatch 2021-11-20T06:01:15.548 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: 2021-11-20T06:01:15.548 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021 2021-11-20T06:01:15.548 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: -11-20T06:01:14.526159+0000 mon.smithi139 (mon. 2021-11-20T06:01:15.549 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: 0) 261 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:15.549 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:01:15.549 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: 20T06:01:14.528453+0000 mon.smithi139 ( 2021-11-20T06:01:15.549 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:15 smithi139 conmon[33501]: mon.0) 262 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:01:15.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: cluster 2021-11-20T06:01:14.186224+0000 mgr.smithi139.dplyxn (mgr.14162) 89 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:15.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11-20T06:01:14.474846+0000 mon.smithi139 (mon.0) 252 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:15.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11-20T06:01:14.478333+0000 mon.smithi139 (mon.0) 253 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:15.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11-20T06:01:14.492348+0000 mon.smithi139 (mon.0) 254 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.139:9093"}]: dispatch 2021-11-20T06:01:15.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11-20T06:01:14.492691+0000 mgr.smithi139.dplyxn (mgr.14162) 90 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.139:9093"}]: dispatch 2021-11-20T06:01:15.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11-20T06:01:14.497360+0000 mon.smithi139 (mon.0) 255 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:15.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11-20T06:01:14.500586+0000 mon.smithi139 (mon.0) 256 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T06:01:15.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11-20T06:01:14.500900+0000 mgr.smithi139.dplyxn (mgr.14162) 91 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-20T06:01:15.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11-20T06:01:14.505452+0000 mon.smithi139 (mon.0) 257 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.139:3000"}]: dispatch 2021-11-20T06:01:15.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11-20T06:01:14.505776+0000 mgr.smithi139.dplyxn (mgr.14162) 92 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.139:3000"}]: dispatch 2021-11-20T06:01:15.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11-20T06:01:14.510074+0000 mon.smithi139 (mon.0) 258 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:15.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11-20T06:01:14.513374+0000 mon.smithi139 (mon.0) 259 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T06:01:15.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11-20T06:01:14.513780+0000 mgr.smithi139.dplyxn (mgr.14162) 93 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-20T06:01:15.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11-20T06:01:14.521006+0000 mon.smithi139 (mon.0) 2021-11-20T06:01:15.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: 260 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.139:9095"}]: dispatch 2021-11-20T06:01:15.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11-20T06:01:14.521363+0000 mgr.smithi139.dplyxn (mgr.14162) 94 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.139:9095"}]: dispatch 2021-11-20T06:01:15.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11-20T06:01:14 2021-11-20T06:01:15.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: .526159+0000 mon.smithi139 (mon.0) 261 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:15.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: audit 2021-11 2021-11-20T06:01:15.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: -20T06:01:14.528453+0000 2021-11-20T06:01:15.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: mon.smithi139 (mon.0) 262 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:01:15.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:15 smithi178 conmon[36600]: 2021-11-20T06:01:16.867 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-11-20T06:01:16.867 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-11-20T06:01:16.868 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2021-11-20T06:01:16.868 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2021-11-20T06:01:16.869 INFO:teuthology.orchestra.run.smithi139.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0120728 s, 869 MB/s 2021-11-20T06:01:16.869 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 2021-11-20T06:01:16.869 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2021-11-20T06:01:17.170 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi139:vg_nvme/lv_2 2021-11-20T06:01:17.921 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:17 smithi139 conmon[33501]: cluster 2021-11-20T06:01:16.186501+0000 mgr.smithi139.dplyxn (mgr.14162) 95 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:17.921 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:17 smithi139 conmon[33501]: audit 2021-11-20T06:01:16.675032+0000 mon.smithi139 (mon.0) 263 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:17.942 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:17 smithi178 conmon[36600]: cluster 2021-11-20T06:01:16.186501+0000 mgr.smithi139.dplyxn (mgr.14162) 95 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:17.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:17 smithi178 conmon[36600]: 2021-11-20T06:01:17.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:17 smithi178 conmon[36600]: audit 2021-11-20T06:01:16.675032+0000 mon.smithi139 (mon.0) 263 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:18.659 INFO:journalctl@ceph.osd.1.smithi139.stdout:Nov 20 06:01:18 smithi139 conmon[55514]: debug 2021-11-20T06:01:18.470+0000 7f4407443080 -1 osd.1 0 log_to_monitors {default=true} 2021-11-20T06:01:18.938 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:18 smithi139 conmon[33501]: audit 2021-11-20T06:01:18.474757+0000 2021-11-20T06:01:18.938 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:18 smithi139 conmon[33501]: mon.smithi178 (mon.1) 2 : audit [INF] from='osd.1 [v2:172.21.15.139:6810/2530906188,v1:172.21.15.139:6811/2530906188]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-11-20T06:01:18.939 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:18 smithi139 conmon[33501]: audit 2021-11-20T06:01:18.475118+0000 mon.smithi139 (mon.0) 264 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-11-20T06:01:18.941 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.951+0000 7f9b5949c700 1 -- 172.21.15.139:0/985506754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b540f7ba0 msgr2=0x7f9b540f7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:01:18.942 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.951+0000 7f9b5949c700 1 --2- 172.21.15.139:0/985506754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b540f7ba0 0x7f9b540f7fc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f9b48004660 tx=0x7f9b48009b30).stop 2021-11-20T06:01:18.942 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.951+0000 7f9b5949c700 1 -- 172.21.15.139:0/985506754 shutdown_connections 2021-11-20T06:01:18.943 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.951+0000 7f9b5949c700 1 --2- 172.21.15.139:0/985506754 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b540f8540 0x7f9b540f46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:18.943 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.951+0000 7f9b5949c700 1 --2- 172.21.15.139:0/985506754 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b540f7ba0 0x7f9b540f7fc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:18.943 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.951+0000 7f9b5949c700 1 -- 172.21.15.139:0/985506754 >> 172.21.15.139:0/985506754 conn(0x7f9b540f0320 msgr2=0x7f9b540f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:01:18.945 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.951+0000 7f9b5949c700 1 -- 172.21.15.139:0/985506754 shutdown_connections 2021-11-20T06:01:18.945 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.951+0000 7f9b5949c700 1 -- 172.21.15.139:0/985506754 wait complete. 2021-11-20T06:01:18.945 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.952+0000 7f9b5949c700 1 Processor -- start 2021-11-20T06:01:18.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.953+0000 7f9b5949c700 1 -- start start 2021-11-20T06:01:18.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.953+0000 7f9b5949c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b540f7ba0 0x7f9b54106e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:01:18.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.953+0000 7f9b5949c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b540f8540 0x7f9b54107320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:01:18.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.953+0000 7f9b5949c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b54107920 con 0x7f9b540f7ba0 2021-11-20T06:01:18.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.953+0000 7f9b5949c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b54108130 con 0x7f9b540f8540 2021-11-20T06:01:18.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.953+0000 7f9b53fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b540f7ba0 0x7f9b54106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:01:18.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.953+0000 7f9b53fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b540f7ba0 0x7f9b54106e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:56972/0 (socket says 172.21.15.139:56972) 2021-11-20T06:01:18.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.953+0000 7f9b53fff700 1 -- 172.21.15.139:0/2677583575 learned_addr learned my addr 172.21.15.139:0/2677583575 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:01:18.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.953+0000 7f9b53fff700 1 -- 172.21.15.139:0/2677583575 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b540f8540 msgr2=0x7f9b54107320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:01:18.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.953+0000 7f9b537fe700 1 --2- 172.21.15.139:0/2677583575 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b540f8540 0x7f9b54107320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:01:18.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.953+0000 7f9b53fff700 1 --2- 172.21.15.139:0/2677583575 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b540f8540 0x7f9b54107320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:18.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.954+0000 7f9b53fff700 1 -- 172.21.15.139:0/2677583575 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b48005040 con 0x7f9b540f7ba0 2021-11-20T06:01:18.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.954+0000 7f9b53fff700 1 --2- 172.21.15.139:0/2677583575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b540f7ba0 0x7f9b54106e30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f9b48009f50 tx=0x7f9b48009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:01:18.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.954+0000 7f9b50ff9700 1 -- 172.21.15.139:0/2677583575 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b48018070 con 0x7f9b540f7ba0 2021-11-20T06:01:18.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.954+0000 7f9b5949c700 1 -- 172.21.15.139:0/2677583575 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b54108350 con 0x7f9b540f7ba0 2021-11-20T06:01:18.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.954+0000 7f9b50ff9700 1 -- 172.21.15.139:0/2677583575 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b48013410 con 0x7f9b540f7ba0 2021-11-20T06:01:18.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.954+0000 7f9b5949c700 1 -- 172.21.15.139:0/2677583575 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b541088b0 con 0x7f9b540f7ba0 2021-11-20T06:01:18.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.954+0000 7f9b50ff9700 1 -- 172.21.15.139:0/2677583575 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b4801c620 con 0x7f9b540f7ba0 2021-11-20T06:01:18.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.955+0000 7f9b5949c700 1 -- 172.21.15.139:0/2677583575 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b54059100 con 0x7f9b540f7ba0 2021-11-20T06:01:18.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.955+0000 7f9b50ff9700 1 -- 172.21.15.139:0/2677583575 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9b4801d420 con 0x7f9b540f7ba0 2021-11-20T06:01:18.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.955+0000 7f9b50ff9700 1 --2- 172.21.15.139:0/2677583575 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9b3c05c300 0x7f9b3c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:01:18.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.956+0000 7f9b50ff9700 1 -- 172.21.15.139:0/2677583575 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2018+0+0 (secure 0 0 0) 0x7f9b48077020 con 0x7f9b540f7ba0 2021-11-20T06:01:18.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.956+0000 7f9b537fe700 1 --2- 172.21.15.139:0/2677583575 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9b3c05c300 0x7f9b3c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:01:18.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.956+0000 7f9b537fe700 1 --2- 172.21.15.139:0/2677583575 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9b3c05c300 0x7f9b3c05e7c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f9b440049e0 tx=0x7f9b44009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:01:18.952 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:18 smithi178 conmon[36600]: audit 2021-11-20T06:01:18.474757+0000 mon.smithi178 2021-11-20T06:01:18.952 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:18 smithi178 conmon[36600]: (mon.1) 2 : audit [INF] from='osd.1 [v2:172.21.15.139:6810/2530906188,v1:172.21.15.139:6811/2530906188]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-11-20T06:01:18.952 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:18 smithi178 conmon[36600]: audit 2021-11-20T06:01:18.475118 2021-11-20T06:01:18.952 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:18 smithi178 conmon[36600]: +0000 mon.smithi139 (mon.0) 264 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-11-20T06:01:18.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:18.959+0000 7f9b50ff9700 1 -- 172.21.15.139:0/2677583575 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b4804aa00 con 0x7f9b540f7ba0 2021-11-20T06:01:19.130 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:19.143+0000 7f9b5949c700 1 -- 172.21.15.139:0/2677583575 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b540fece0 con 0x7f9b3c05c300 2021-11-20T06:01:19.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: cluster 2021-11-20T06:01:18.186770+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:01:19.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: mgr.14162) 96 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:19.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: audit 2021-11-20T06:01:18.681579+0000 mon.smithi139 ( 2021-11-20T06:01:19.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: mon.0) 265 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-11-20T06:01:19.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: cluster 2021-11-20T06:01:18.681669+0000 mon.smithi139 (mon.0 2021-11-20T06:01:19.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: ) 266 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-11-20T06:01:19.948 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: audit 2021-11 2021-11-20T06:01:19.948 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: -20T06:01:18.681771+0000 mon.smithi139 (mon.0) 2021-11-20T06:01:19.948 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: 267 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T06:01:19.948 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: audit 2021-11-20T 2021-11-20T06:01:19.949 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: 06:01:18.682416+0000 mon.smithi178 (mon.1) 3 : audit 2021-11-20T06:01:19.949 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: [INF] from='osd.1 [v2:172.21.15.139:6810/2530906188,v1:172.21.15.139:6811/2530906188]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2021-11-20T06:01:19.949 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: audit 2021-11-20T06:01:18. 2021-11-20T06:01:19.949 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: 682691+0000 mon.smithi139 (mon.0) 268 : audit [INF] 2021-11-20T06:01:19.949 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2021-11-20T06:01:19.950 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: audit 2021-11-20T06:01:19.145079 2021-11-20T06:01:19.950 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: +0000 mgr.smithi139.dplyxn (mgr.14162) 97 : audit [DBG] from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:01:19.950 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: 2021-11-20T06:01:19.951 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: audit 2021-11-20T06:01:19.145853 2021-11-20T06:01:19.951 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: +0000 mon.smithi139 (mon.0) 269 : audit [DBG] 2021-11-20T06:01:19.951 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T06:01:19.951 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: cephadm 2021-11-20T06: 2021-11-20T06:01:19.952 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: 01:19.147022+0000 mgr.smithi139.dplyxn (mgr.14162) 98 : 2021-11-20T06:01:19.952 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: cephadm [INF] Applying service osd.None on host smithi139... 2021-11-20T06:01:19.952 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: audit 2021-11-20 2021-11-20T06:01:19.952 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: T06:01:19.147237+0000 mon.smithi139 (mon.0) 270 2021-11-20T06:01:19.952 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T06:01:19.953 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: audit 2021-11-20T 2021-11-20T06:01:19.953 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: 06:01:19.147839+0000 mon.smithi139 (mon.0) 271 : 2021-11-20T06:01:19.953 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:19 smithi178 conmon[36600]: audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:19.973 INFO:journalctl@ceph.osd.1.smithi139.stdout:Nov 20 06:01:19 smithi139 conmon[55514]: debug 2021-11-20T06:01:19.688+0000 7f43fd9b4700 -1 osd.1 0 waiting for initial osdmap 2021-11-20T06:01:19.973 INFO:journalctl@ceph.osd.1.smithi139.stdout:Nov 20 06:01:19 smithi139 conmon[55514]: debug 2021-11-20T06:01:19.692+0000 7f43fa130700 -1 osd.1 13 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T06:01:19.974 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:19 smithi139 conmon[33501]: cluster 2021-11-20T06:01:18.186770+0000 mgr.smithi139.dplyxn (mgr.14162) 96 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:19.974 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:19 smithi139 conmon[33501]: audit 2021-11-20T06:01:18.681579+0000 mon.smithi139 (mon.0) 265 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-11-20T06:01:19.974 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:19 smithi139 conmon[33501]: cluster 2021-11-20T06:01:18.681669+0000 mon.smithi139 (mon.0) 266 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-11-20T06:01:19.974 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:19 smithi139 conmon[33501]: audit 2021-11-20T06:01:18.681771+0000 mon.smithi139 (mon.0) 267 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T06:01:19.975 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:19 smithi139 conmon[33501]: audit 2021-11-20T06:01:18.682416+0000 mon.smithi178 (mon.1) 3 : audit [INF] from='osd.1 [v2:172.21.15.139:6810/2530906188,v1:172.21.15.139:6811/2530906188]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2021-11-20T06:01:19.975 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:19 smithi139 conmon[33501]: audit 2021-11-20T06:01:18.682691+0000 mon.smithi139 (mon.0) 268 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2021-11-20T06:01:19.975 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:19 smithi139 conmon[33501]: audit 2021-11-20T06:01:19.145079+0000 mgr.smithi139.dplyxn (mgr.14162) 97 : audit [DBG] from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:01:19.975 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:19 smithi139 conmon[33501]: audit 2021-11-20T06:01:19.145853+0000 mon.smithi139 (mon.0) 269 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T06:01:19.976 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:19 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:19.147022+0000 mgr.smithi139.dplyxn (mgr.14162) 98 : cephadm [INF] Applying service osd.None on host smithi139... 2021-11-20T06:01:19.976 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:19 smithi139 conmon[33501]: audit 2021-11-20T06:01:19.147237+0000 mon.smithi139 (mon.0) 270 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T06:01:19.976 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:19 smithi139 conmon[33501]: audit 2021-11-20T06:01:19.147839+0000 mon.smithi139 (mon.0) 271 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:20.851 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:20 smithi139 conmon[33501]: audit 2021-11-20T06:01:19.685007+0000 mon.smithi139 2021-11-20T06:01:20.852 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:20 smithi139 conmon[33501]: (mon.0) 272 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2021-11-20T06:01:20.852 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:20 smithi139 conmon[33501]: cluster 2021-11-20T06:01:19.685075+0000 mon.smithi139 (mon.0) 273 : cluster [DBG] osdmap e13: 2 total, 1 up, 2 in 2021-11-20T06:01:20.852 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:20 smithi139 conmon[33501]: audit 2021-11-20T06:01:19.686032+0000 mon.smithi139 (mon.0) 274 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T06:01:20.852 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:20 smithi139 conmon[33501]: audit 2021-11-20T06:01:19.689922+0000 mon.smithi139 (mon.0) 275 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T06:01:20.853 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:20 smithi139 conmon[33501]: audit 2021-11-20T06:01:20.113267+0000 mon.smithi139 (mon.0) 276 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:20.942 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:20 smithi178 conmon[36600]: audit 2021-11-20T06:01:19.685007+0000 mon.smithi139 ( 2021-11-20T06:01:20.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:20 smithi178 conmon[36600]: mon.0) 272 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2021-11-20T06:01:20.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:20 smithi178 conmon[36600]: cluster 2021-11-20T06:01:19.685075+0000 mon.smithi139 (mon 2021-11-20T06:01:20.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:20 smithi178 conmon[36600]: .0) 273 : cluster [DBG] osdmap e13: 2 total, 1 up, 2 in 2021-11-20T06:01:20.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:20 smithi178 conmon[36600]: audit 2021-11-20T06:01:20.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:20 smithi178 conmon[36600]: 2021-11-20T06:01:19.686032+0000 mon.smithi139 (mon.0) 274 : audit [DBG] 2021-11-20T06:01:20.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:20 smithi178 conmon[36600]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T06:01:20.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:20 smithi178 conmon[36600]: audit 2021-11-20T06:01:19. 2021-11-20T06:01:20.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:20 smithi178 conmon[36600]: 689922+0000 mon.smithi139 (mon.0) 275 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T06:01:20.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:20 smithi178 conmon[36600]: 2021-11-20T06:01:20.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:20 smithi178 conmon[36600]: audit 2021-11-20T06:01:20.113267 2021-11-20T06:01:20.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:20 smithi178 conmon[36600]: +0000 mon.smithi139 (mon.0) 276 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:21.942 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: cluster 2021-11-20T06:01:19.499020+0000 osd.1 (osd.1) 1 : 2021-11-20T06:01:21.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: cluster [DBG] purged_snaps scrub starts 2021-11-20T06:01:21.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: cluster 2021-11-20T06:01:19.499147+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T06:01:21.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: cluster 2021-11-20T06:01:21.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: 2021-11-20T06:01:20.187235+0000 mgr.smithi139.dplyxn (mgr.14162) 99 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:21.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: audit 2021-11-20T06:01:20. 2021-11-20T06:01:21.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: 688693+0000 mon.smithi139 (mon.0) 277 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T06:01:21.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: 2021-11-20T06:01:21.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: cluster 2021-11-20T06:01:20.694716+0000 2021-11-20T06:01:21.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: mon.smithi139 (mon.0) 278 : cluster [INF] osd.1 [v2:172.21.15.139:6810/2530906188,v1:172.21.15.139:6811/2530906188] boot 2021-11-20T06:01:21.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: cluster 2021-11-20T06:01:21.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: 2021-11-20T06:01:20.694764+0000 mon.smithi139 (mon. 2021-11-20T06:01:21.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: 0) 279 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-11-20T06:01:21.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: audit 2021- 2021-11-20T06:01:21.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: 11-20T06:01:20.695179+0000 mon.smithi139 (mon 2021-11-20T06:01:21.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:21 smithi178 conmon[36600]: .0) 280 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T06:01:22.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:21 smithi139 conmon[33501]: cluster 2021-11-20T06:01:19.499020+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T06:01:22.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:21 smithi139 conmon[33501]: cluster 2021-11-20T06:01:19.499147+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T06:01:22.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:21 smithi139 conmon[33501]: cluster 2021-11-20T06:01:20.187235+0000 mgr.smithi139.dplyxn (mgr.14162) 99 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-20T06:01:22.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:21 smithi139 conmon[33501]: audit 2021-11-20T06:01:20.688693+0000 mon.smithi139 (mon.0) 277 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T06:01:22.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:21 smithi139 conmon[33501]: cluster 2021-11-20T06:01:20.694716+0000 mon.smithi139 (mon.0) 278 : cluster [INF] osd.1 [v2:172.21.15.139:6810/2530906188,v1:172.21.15.139:6811/2530906188] boot 2021-11-20T06:01:22.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:21 smithi139 conmon[33501]: cluster 2021-11-20T06:01:20.694764+0000 mon.smithi139 (mon.0) 279 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-11-20T06:01:22.030 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:21 smithi139 conmon[33501]: audit 2021-11-20T06:01:20.695179+0000 mon.smithi139 (mon.0) 280 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-20T06:01:23.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:22 smithi139 conmon[33501]: audit 2021-11-20T06:01:21.892668+0000 mon.smithi139 (mon.0) 281 : 2021-11-20T06:01:23.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:22 smithi139 conmon[33501]: audit [INF] from='client.? 172.21.15.139:0/2183883334' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80cfe585-e283-45a4-9dd7-f328ec00d21c"}]: dispatch 2021-11-20T06:01:23.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:22 smithi139 conmon[33501]: audit 2021-11-20T06:01:21.896431+0000 mon.smithi139 (mon.0) 282 : audit [INF] from='client.? 172.21.15.139:0/2183883334' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80cfe585-e283-45a4-9dd7-f328ec00d21c"}]': finished 2021-11-20T06:01:23.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:22 smithi139 conmon[33501]: cluster 2021-11-20T06:01:21.896473+0000 mon.smithi139 (mon.0) 283 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-11-20T06:01:23.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:22 smithi139 conmon[33501]: audit 2021-11-20T06:01:21.896556+0000 mon.smithi139 (mon.0) 284 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T06:01:23.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:22 smithi139 conmon[33501]: audit 2021-11-20T06:01:22.298733+0000 mon.smithi139 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.139:0/1891088284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T06:01:23.192 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:22 smithi178 conmon[36600]: audit 2021-11-20T06:01:21.892668+0000 mon.smithi139 (mon.0) 2021-11-20T06:01:23.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:22 smithi178 conmon[36600]: 281 : audit [INF] from='client.? 172.21.15.139:0/2183883334' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80cfe585-e283-45a4-9dd7-f328ec00d21c"}]: dispatch 2021-11-20T06:01:23.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:22 smithi178 conmon[36600]: audit 2021-11-20T06:01:21.896431+0000 mon.smithi139 (mon.0) 282 : audit 2021-11-20T06:01:23.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:22 smithi178 conmon[36600]: [INF] from='client.? 172.21.15.139:0/2183883334' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80cfe585-e283-45a4-9dd7-f328ec00d21c"}]': finished 2021-11-20T06:01:23.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:22 smithi178 conmon[36600]: cluster 2021-11-20T 2021-11-20T06:01:23.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:22 smithi178 conmon[36600]: 06:01:21.896473+0000 mon.smithi139 (mon.0) 283 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-11-20T06:01:23.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:22 smithi178 conmon[36600]: audit 2021-11-20T06:01:23.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:22 smithi178 conmon[36600]: 2021-11-20T06:01:21.896556+0000 mon.smithi139 (mon. 2021-11-20T06:01:23.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:22 smithi178 conmon[36600]: 0) 284 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T06:01:23.195 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:22 smithi178 conmon[36600]: audit 2021-11- 2021-11-20T06:01:23.195 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:22 smithi178 conmon[36600]: 20T06:01:22.298733+0000 mon.smithi139 (mon.0 2021-11-20T06:01:23.195 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:22 smithi178 conmon[36600]: ) 285 : audit [DBG] from='client.? 172.21.15.139:0/1891088284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T06:01:24.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:24 smithi178 conmon[36600]: cluster 2021-11-20T06:01:22.187510+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:01:24.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:24 smithi178 conmon[36600]: 14162) 100 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:24.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:24 smithi139 conmon[33501]: cluster 2021-11-20T06:01:22.187510+0000 mgr.smithi139.dplyxn (mgr.14162) 100 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:26.190 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:26 smithi139 conmon[33501]: cluster 2021-11-20T06:01:24.187788+0000 mgr.smithi139.dplyxn (mgr.14162) 101 2021-11-20T06:01:26.190 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:26 smithi139 conmon[33501]: : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:26.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:26 smithi178 conmon[36600]: cluster 2021-11-20T06:01:24.187788+0000 2021-11-20T06:01:26.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:26 smithi178 conmon[36600]: mgr.smithi139.dplyxn (mgr.14162) 101 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:28.413 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:28 smithi139 conmon[33501]: cluster 2021-11-20T06:01:26.188025+0000 mgr.smithi139.dplyxn (mgr.14162) 102 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:28.413 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:28 smithi139 conmon[33501]: audit 2021-11-20T06:01:27.908019+0000 mon.smithi139 (mon.0) 286 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:28.414 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:28 smithi139 conmon[33501]: audit 2021-11-20T06:01:27.911425+0000 mon.smithi139 (mon.0) 287 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:28.414 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:28 smithi139 conmon[33501]: audit 2021-11-20T06:01:27.913575+0000 mon.smithi139 (mon.0) 288 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:01:28.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:28 smithi178 conmon[36600]: cluster 2021-11-20T06:01:26.188025+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:01:28.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:28 smithi178 conmon[36600]: ) 102 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:28.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:28 smithi178 conmon[36600]: audit 2021-11-20T06:01:27.908019+0000 mon.smithi139 (mon.0) 286 : audit [INF] 2021-11-20T06:01:28.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:28 smithi178 conmon[36600]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:28.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:28 smithi178 conmon[36600]: audit 2021-11- 2021-11-20T06:01:28.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:28 smithi178 conmon[36600]: 20T06:01:27.911425+0000 mon.smithi139 (mon.0) 287 : audit 2021-11-20T06:01:28.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:28 smithi178 conmon[36600]: [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:28.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:28 smithi178 conmon[36600]: audit 2021-11-20T06:01 2021-11-20T06:01:28.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:28 smithi178 conmon[36600]: :27.913575+0000 mon.smithi139 (mon.0) 288 : 2021-11-20T06:01:28.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:28 smithi178 conmon[36600]: audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:01:29.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:29 smithi178 conmon[36600]: cephadm 2021-11-20T06:01:27.903328+0000 mgr.smithi139.dplyxn (mgr 2021-11-20T06:01:29.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:29 smithi178 conmon[36600]: .14162) 103 : cephadm [INF] Detected new or changed devices on smithi139 2021-11-20T06:01:29.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:29 smithi178 conmon[36600]: audit 2021-11-20T06:01:28.714202+0000 mon.smithi139 (mon.0) 289 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-11-20T06:01:29.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:29 smithi178 conmon[36600]: audit 2021-11-20T06:01:28.714787+0000 mon.smithi139 (mon.0) 290 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:29.451 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:29 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:27.903328+0000 mgr.smithi139.dplyxn (mgr.14162) 103 : cephadm [INF] Detected new or changed devices on smithi139 2021-11-20T06:01:29.451 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:29 smithi139 conmon[33501]: audit 2021-11-20T06:01:28.714202+0000 mon.smithi139 (mon.0) 289 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-11-20T06:01:29.452 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:29 smithi139 conmon[33501]: audit 2021-11-20T06:01:28.714787+0000 mon.smithi139 (mon.0) 290 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:30.271 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:30 smithi139 conmon[33501]: cluster 2021-11-20T06:01:28.188270+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:01:30.271 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:30 smithi139 conmon[33501]: 14162) 104 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:30.271 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:30 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:28.715227+0000 mgr.smithi139.dplyxn (mgr.14162) 105 : cephadm [INF] Deploying daemon osd.2 on smithi139 2021-11-20T06:01:30.272 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:30 smithi139 conmon[33501]: audit 2021-11-20T06:01:30.061575+0000 mon.smithi139 (mon.0) 291 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:30.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:30 smithi178 conmon[36600]: cluster 2021-11-20T06:01:28.188270+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:01:30.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:30 smithi178 conmon[36600]: ) 104 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:30.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:30 smithi178 conmon[36600]: cephadm 2021-11-20T06:01:28.715227+0000 mgr.smithi139.dplyxn (mgr.14162) 105 : cephadm [INF] Deploying daemon osd.2 on smithi139 2021-11-20T06:01:30.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:30 smithi178 conmon[36600]: audit 2021-11-20T06:01:30.061575+0000 mon.smithi139 (mon.0 2021-11-20T06:01:30.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:30 smithi178 conmon[36600]: ) 291 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:32.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:32 smithi139 conmon[33501]: cluster 2021-11-20T06:01:30.188524+0000 mgr.smithi139.dplyxn (mgr.14162) 106 : 2021-11-20T06:01:32.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:32 smithi139 conmon[33501]: cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:32.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:32 smithi139 conmon[33501]: audit 2021-11-20T06:01:31.249722+0000 mon.smithi139 (mon.0) 292 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:01:32.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:32 smithi139 conmon[33501]: audit 2021-11-20T06:01:31.249989+0000 mon.smithi139 (mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:01:32.307 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 2 on host 'smithi139' 2021-11-20T06:01:32.308 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:32.319+0000 7f9b50ff9700 1 -- 172.21.15.139:0/2677583575 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9b540fece0 con 0x7f9b3c05c300 2021-11-20T06:01:32.311 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:32.321+0000 7f9b5949c700 1 -- 172.21.15.139:0/2677583575 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9b3c05c300 msgr2=0x7f9b3c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:01:32.311 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:32.321+0000 7f9b5949c700 1 --2- 172.21.15.139:0/2677583575 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9b3c05c300 0x7f9b3c05e7c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f9b440049e0 tx=0x7f9b44009040).stop 2021-11-20T06:01:32.311 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:32.322+0000 7f9b5949c700 1 -- 172.21.15.139:0/2677583575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b540f7ba0 msgr2=0x7f9b54106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:01:32.311 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:32.322+0000 7f9b5949c700 1 --2- 172.21.15.139:0/2677583575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b540f7ba0 0x7f9b54106e30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f9b48009f50 tx=0x7f9b48009b30).stop 2021-11-20T06:01:32.312 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:32.322+0000 7f9b5949c700 1 -- 172.21.15.139:0/2677583575 shutdown_connections 2021-11-20T06:01:32.312 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:32.322+0000 7f9b5949c700 1 --2- 172.21.15.139:0/2677583575 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9b3c05c300 0x7f9b3c05e7c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:32.312 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:32.322+0000 7f9b5949c700 1 --2- 172.21.15.139:0/2677583575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b540f7ba0 0x7f9b54106e30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:32.312 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:32.322+0000 7f9b5949c700 1 --2- 172.21.15.139:0/2677583575 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b540f8540 0x7f9b54107320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:32.313 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:32.322+0000 7f9b5949c700 1 -- 172.21.15.139:0/2677583575 >> 172.21.15.139:0/2677583575 conn(0x7f9b540f0320 msgr2=0x7f9b540fb430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:01:32.313 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:32.322+0000 7f9b5949c700 1 -- 172.21.15.139:0/2677583575 shutdown_connections 2021-11-20T06:01:32.313 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:01:32.322+0000 7f9b5949c700 1 -- 172.21.15.139:0/2677583575 wait complete. 2021-11-20T06:01:32.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:32 smithi178 conmon[36600]: cluster 2021-11-20T06:01:30.188524+0000 mgr.smithi139.dplyxn (mgr.14162) 106 : cluster [DBG] 2021-11-20T06:01:32.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:32 smithi178 conmon[36600]: pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:32.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:32 smithi178 conmon[36600]: audit 2021-11-20T06:01:31.249722+0000 mon.smithi139 (mon.0) 292 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:01:32.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:32 smithi178 conmon[36600]: audit 2021-11-20T06:01:32.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:32 smithi178 conmon[36600]: 2021-11-20T06:01:31.249989+0000 mon.smithi139 (mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:01:32.725 DEBUG:teuthology.orchestra.run.smithi139:osd.2> sudo journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.2.service 2021-11-20T06:01:32.727 INFO:tasks.cephadm:Deploying osd.3 on smithi178 with /dev/vg_nvme/lv_4... 2021-11-20T06:01:32.728 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-11-20T06:01:32.769 INFO:journalctl@ceph.osd.2.smithi139.stdout:-- Logs begin at Sat 2021-11-20 05:47:58 UTC. -- 2021-11-20T06:01:33.437 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:33 smithi178 conmon[36600]: cluster 2021-11-20T06:01:32.188830+0000 2021-11-20T06:01:33.438 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:33 smithi178 conmon[36600]: mgr.smithi139.dplyxn (mgr.14162) 107 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:33.439 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:33 smithi178 conmon[36600]: audit 2021-11-20T06:01:32.313889+0000 mon.smithi139 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:33.439 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:33 smithi178 conmon[36600]: audit 2021-11-20T06:01:32.320822+0000 mon.smithi139 (mon.0) 295 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:33.727 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:33 smithi139 conmon[33501]: cluster 2021-11-20T06:01:32.188830+0000 mgr.smithi139.dplyxn (mgr.14162) 107 : cluster 2021-11-20T06:01:33.727 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:33 smithi139 conmon[33501]: [DBG] pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:33.728 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:33 smithi139 conmon[33501]: audit 2021-11-20T06:01:32.313889+0000 mon.smithi139 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:33.728 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:33 smithi139 conmon[33501]: audit 2021-11-20T06:01:32.320822+0000 mon.smithi139 (mon.0) 295 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:34.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:34 smithi178 conmon[36600]: audit 2021-11-20T06:01:33.324482+0000 mon.smithi139 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:34.611 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:34 smithi139 conmon[33501]: audit 2021-11-20T06:01:33.324482+0000 mon.smithi139 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:34.700 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-11-20T06:01:34.701 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-11-20T06:01:34.701 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2021-11-20T06:01:34.701 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2021-11-20T06:01:34.701 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00826785 s, 1.3 GB/s 2021-11-20T06:01:34.702 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2021-11-20T06:01:35.053 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi178:vg_nvme/lv_4 2021-11-20T06:01:35.570 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:35 smithi178 conmon[36600]: cluster 2021-11-20T06:01:34.189128+0000 mgr.smithi139.dplyxn (mgr.14162) 108 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:35.632 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:35 smithi139 conmon[33501]: cluster 2021-11-20T06:01:34.189128+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:01:35.633 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:35 smithi139 conmon[33501]: 14162) 108 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:36.277 INFO:journalctl@ceph.osd.2.smithi139.stdout:Nov 20 06:01:36 smithi139 conmon[61885]: debug 2021-11-20T06:01:36.215+0000 7f0a64967080 -1 osd.2 0 log_to_monitors {default=true} 2021-11-20T06:01:36.655 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.669+0000 7f75a7dbe700 1 -- 172.21.15.178:0/871500197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a00f55d0 msgr2=0x7f75a00f59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:01:36.655 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.669+0000 7f75a7dbe700 1 --2- 172.21.15.178:0/871500197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a00f55d0 0x7f75a00f59f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f759c004660 tx=0x7f759c009b30).stop 2021-11-20T06:01:36.656 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.669+0000 7f75a7dbe700 1 -- 172.21.15.178:0/871500197 shutdown_connections 2021-11-20T06:01:36.656 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.669+0000 7f75a7dbe700 1 --2- 172.21.15.178:0/871500197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a00f6870 0x7f75a00f6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:36.656 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.669+0000 7f75a7dbe700 1 --2- 172.21.15.178:0/871500197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a00f55d0 0x7f75a00f59f0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:36.656 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.669+0000 7f75a7dbe700 1 -- 172.21.15.178:0/871500197 >> 172.21.15.178:0/871500197 conn(0x7f75a00f0d70 msgr2=0x7f75a00f3190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:01:36.657 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.669+0000 7f75a7dbe700 1 -- 172.21.15.178:0/871500197 shutdown_connections 2021-11-20T06:01:36.657 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.669+0000 7f75a7dbe700 1 -- 172.21.15.178:0/871500197 wait complete. 2021-11-20T06:01:36.657 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:36 smithi178 conmon[36600]: audit 2021-11-20T06:01:36.219706+0000 mon.smithi139 (mon.0) 297 : audit 2021-11-20T06:01:36.658 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:36 smithi178 conmon[36600]: [INF] from='osd.2 [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-11-20T06:01:36.660 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.669+0000 7f75a7dbe700 1 Processor -- start 2021-11-20T06:01:36.660 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.669+0000 7f75a7dbe700 1 -- start start 2021-11-20T06:01:36.660 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75a7dbe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a00f55d0 0x7f75a0106f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:01:36.661 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75a7dbe700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a00f6870 0x7f75a0107460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:01:36.661 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75a7dbe700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a0107a10 con 0x7f75a00f55d0 2021-11-20T06:01:36.661 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75a7dbe700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a0107b50 con 0x7f75a00f6870 2021-11-20T06:01:36.661 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75a5359700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a00f6870 0x7f75a0107460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:01:36.662 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75a5359700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a00f6870 0x7f75a0107460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:50120/0 (socket says 172.21.15.178:50120) 2021-11-20T06:01:36.662 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75a5359700 1 -- 172.21.15.178:0/632040460 learned_addr learned my addr 172.21.15.178:0/632040460 (peer_addr_for_me v2:172.21.15.178:0/0) 2021-11-20T06:01:36.662 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75a5b5a700 1 --2- 172.21.15.178:0/632040460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a00f55d0 0x7f75a0106f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:01:36.662 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75a5359700 1 -- 172.21.15.178:0/632040460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a00f55d0 msgr2=0x7f75a0106f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:01:36.662 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75a5359700 1 --2- 172.21.15.178:0/632040460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a00f55d0 0x7f75a0106f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:36.663 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75a5359700 1 -- 172.21.15.178:0/632040460 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c005040 con 0x7f75a00f6870 2021-11-20T06:01:36.663 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75a5359700 1 --2- 172.21.15.178:0/632040460 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a00f6870 0x7f75a0107460 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f7590002700 tx=0x7f7590009e00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:01:36.663 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75967fc700 1 -- 172.21.15.178:0/632040460 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7590009320 con 0x7f75a00f6870 2021-11-20T06:01:36.663 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75a7dbe700 1 -- 172.21.15.178:0/632040460 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a01084f0 con 0x7f75a00f6870 2021-11-20T06:01:36.664 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75967fc700 1 -- 172.21.15.178:0/632040460 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f759000f040 con 0x7f75a00f6870 2021-11-20T06:01:36.664 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75967fc700 1 -- 172.21.15.178:0/632040460 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7590014820 con 0x7f75a00f6870 2021-11-20T06:01:36.664 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.670+0000 7f75a7dbe700 1 -- 172.21.15.178:0/632040460 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a0108ab0 con 0x7f75a00f6870 2021-11-20T06:01:36.664 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.671+0000 7f75a7dbe700 1 -- 172.21.15.178:0/632040460 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a0045ad0 con 0x7f75a00f6870 2021-11-20T06:01:36.664 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.672+0000 7f75967fc700 1 -- 172.21.15.178:0/632040460 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f75900077b0 con 0x7f75a00f6870 2021-11-20T06:01:36.665 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.672+0000 7f75967fc700 1 --2- 172.21.15.178:0/632040460 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f758c05c2a0 0x7f758c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:01:36.665 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.672+0000 7f75967fc700 1 -- 172.21.15.178:0/632040460 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2450+0+0 (secure 0 0 0) 0x7f75900756d0 con 0x7f75a00f6870 2021-11-20T06:01:36.665 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.672+0000 7f75a5b5a700 1 --2- 172.21.15.178:0/632040460 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f758c05c2a0 0x7f758c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:01:36.665 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.672+0000 7f75a5b5a700 1 --2- 172.21.15.178:0/632040460 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f758c05c2a0 0x7f758c05e760 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f759c005010 tx=0x7f759c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:01:36.666 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.676+0000 7f75967fc700 1 -- 172.21.15.178:0/632040460 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7590048f00 con 0x7f75a00f6870 2021-11-20T06:01:36.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:36 smithi139 conmon[33501]: audit 2021-11-20T06:01:36.219706+0000 mon.smithi139 (mon.0) 297 : audit [INF] from='osd.2 [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-11-20T06:01:36.843 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:36.857+0000 7f75a7dbe700 1 -- 172.21.15.178:0/632040460 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f75a0108d30 con 0x7f758c05c2a0 2021-11-20T06:01:37.640 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:37 smithi178 conmon[36600]: cluster 2021-11-20T06:01:36.189404+0000 mgr.smithi139.dplyxn (mgr 2021-11-20T06:01:37.640 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:37 smithi178 conmon[36600]: .14162) 109 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:37.641 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:37 smithi178 conmon[36600]: audit 2021-11-20T06:01:36.335255+0000 mon.smithi139 (mon.0) 298 : audit [INF] from='osd.2 [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-11-20T06:01:37.641 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:37 smithi178 conmon[36600]: cluster 2021-11-20T06:01:36.335325+0000 mon.smithi139 (mon.0) 299 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-11-20T06:01:37.641 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:37 smithi178 conmon[36600]: audit 2021-11-20T06:01:36.335425+0000 mon.smithi139 (mon.0) 300 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T06:01:37.641 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:37 smithi178 conmon[36600]: audit 2021-11-20T06:01:36.335754+0000 mon.smithi139 (mon.0) 301 : audit [INF] from='osd.2 [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2021-11-20T06:01:37.642 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:37 smithi178 conmon[36600]: audit 2021-11-20T06:01:36.423271+0000 mon.smithi139 (mon.0) 302 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:37.642 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:37 smithi178 conmon[36600]: audit 2021-11-20T06:01:36.426146+0000 mon.smithi139 (mon.0) 303 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:37.642 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:37 smithi178 conmon[36600]: audit 2021-11-20T06:01:36.428318+0000 mon.smithi139 (mon.0) 304 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:01:37.642 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:37 smithi178 conmon[36600]: audit 2021-11-20T06:01:36.859420+0000 mgr.smithi139.dplyxn (mgr.14162) 110 : audit [DBG] from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:01:37.643 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:37 smithi178 conmon[36600]: audit 2021-11-20T06:01:36.860224+0000 mon.smithi139 (mon.0) 305 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T06:01:37.643 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:37 smithi178 conmon[36600]: cephadm 2021-11-20T06:01:36.861843+0000 mgr.smithi139.dplyxn (mgr.14162) 111 : cephadm [INF] Applying service osd.None on host smithi178... 2021-11-20T06:01:37.643 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:37 smithi178 conmon[36600]: audit 2021-11-20T06:01:36.862132+0000 mon.smithi139 (mon.0) 306 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T06:01:37.644 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:37 smithi178 conmon[36600]: audit 2021-11-20T06:01:36.862804+0000 mon.smithi139 (mon.0) 307 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:37.778 INFO:journalctl@ceph.osd.2.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[61885]: debug 2021-11-20T06:01:37.337+0000 7f0a5aed8700 -1 osd.2 0 waiting for initial osdmap 2021-11-20T06:01:37.778 INFO:journalctl@ceph.osd.2.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[61885]: debug 2021-11-20T06:01:37.343+0000 7f0a56e53700 -1 osd.2 17 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T06:01:37.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: cluster 2021-11-20T06:01:36.189404 2021-11-20T06:01:37.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: +0000 mgr.smithi139.dplyxn (mgr.14162) 109 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:37.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: audit 2021-11-20T06:01:36.335255+0000 mon.smithi139 (mon.0) 298 : audit [INF] from='osd.2 [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-11-20T06:01:37.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: cluster 2021-11-20T06:01:36.335325+0000 mon.smithi139 (mon.0) 299 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-11-20T06:01:37.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: audit 2021-11-20T06:01:36.335425+0000 mon.smithi139 (mon.0) 300 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T06:01:37.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: audit 2021-11-20T06:01:36.335754+0000 mon.smithi139 (mon.0) 301 : audit 2021-11-20T06:01:37.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: [INF] from='osd.2 [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2021-11-20T06:01:37.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: audit 2021-11-20T06:01:36.423271+0000 mon.smithi139 2021-11-20T06:01:37.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: (mon.0) 302 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:37.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: audit 2021-11-20 2021-11-20T06:01:37.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: T06:01:36.426146+0000 mon.smithi139 2021-11-20T06:01:37.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: (mon.0) 303 : audit [INF] 2021-11-20T06:01:37.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:37.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: audit 2021-11-20T06:01:37.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: 2021-11 2021-11-20T06:01:37.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: -20T06:01:36.428318 2021-11-20T06:01:37.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 304 : 2021-11-20T06:01:37.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:01:37.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: audit 2021 2021-11-20T06:01:37.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: -11-20T06:01:36 2021-11-20T06:01:37.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: .859420+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:01:37.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: ) 110 : audit [DBG] from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:01:37.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: audit 2021 2021-11-20T06:01:37.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: -11-20T06:01:36. 2021-11-20T06:01:37.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: 860224+0000 mon.smithi139 (mon.0) 305 2021-11-20T06:01:37.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T06:01:37.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: cephadm 2021 2021-11-20T06:01:37.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: -11-20T06:01:36. 2021-11-20T06:01:37.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: 861843+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:01:37.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: 111 : cephadm [INF] Applying service osd.None on host smithi178... 2021-11-20T06:01:37.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: audit 2021 2021-11-20T06:01:37.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: -11-20T06:01:36 2021-11-20T06:01:37.786 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: .862132+0000 mon.smithi139 (mon.0 2021-11-20T06:01:37.786 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: ) 306 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T06:01:37.786 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: audit 2021- 2021-11-20T06:01:37.786 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: 11-20T06:01:36.862804 2021-11-20T06:01:37.787 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 307 2021-11-20T06:01:37.787 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:37 smithi139 conmon[33501]: : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:38.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:38 smithi178 conmon[36600]: audit 2021-11-20T06:01:37.335743+0000 mon.smithi139 (mon.0) 308 : audit [INF] from='osd.2 [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2021-11-20T06:01:38.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:38 smithi178 conmon[36600]: cluster 2021-11-20T06:01:37.335800+0000 mon.smithi139 (mon.0) 309 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-11-20T06:01:38.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:38 smithi178 conmon[36600]: audit 2021-11-20T06:01:37.336170+0000 mon.smithi139 (mon.0) 310 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T06:01:38.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:38 smithi178 conmon[36600]: audit 2021-11-20T06:01:37.340569+0000 mon.smithi139 (mon.0) 311 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T06:01:38.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:38 smithi139 conmon[33501]: audit 2021-11-20T06:01:37.335743+0000 2021-11-20T06:01:38.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:38 smithi139 conmon[33501]: mon.smithi139 (mon.0) 308 : audit [INF] from='osd.2 [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2021-11-20T06:01:38.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:38 smithi139 conmon[33501]: cluster 2021-11-20T06:01:37.335800+0000 mon.smithi139 (mon.0) 309 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-11-20T06:01:38.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:38 smithi139 conmon[33501]: audit 2021-11-20T06:01:37.336170+0000 mon.smithi139 (mon.0) 310 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T06:01:38.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:38 smithi139 conmon[33501]: audit 2021-11-20T06:01:37.340569+0000 mon.smithi139 (mon.0) 311 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T06:01:39.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:39 smithi178 conmon[36600]: cluster 2021-11-20T06:01:37.169425+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T06:01:39.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:39 smithi178 conmon[36600]: cluster 2021- 2021-11-20T06:01:39.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:39 smithi178 conmon[36600]: 11-20T06:01:37.169635+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T06:01:39.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:39 smithi178 conmon[36600]: cluster 2021-11-20T06:01:38.189863+0000 mgr.smithi139.dplyxn (mgr.14162) 112 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:39.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:39 smithi178 conmon[36600]: audit 2021-11-20T06:01:38.340927+0000 mon.smithi139 (mon.0) 312 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T06:01:39.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:39 smithi178 conmon[36600]: cluster 2021-11-20T06:01:38.345724+0000 mon.smithi139 (mon.0) 313 : cluster [INF] osd.2 [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903] boot 2021-11-20T06:01:39.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:39 smithi178 conmon[36600]: cluster 2021-11-20T06:01:38.345755+0000 mon.smithi139 (mon.0) 314 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-11-20T06:01:39.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:39 smithi178 conmon[36600]: audit 2021-11-20T06:01:38.346011+0000 mon.smithi139 (mon.0) 315 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T06:01:39.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:39 smithi178 conmon[36600]: audit 2021-11-20T06:01:38.916180+0000 mon.smithi139 (mon.0) 316 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:39.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:39 smithi178 conmon[36600]: audit 2021-11-20T06:01:39.032786+0000 mon.smithi178 (mon.1) 4 : audit [INF] from='client.? 172.21.15.178:0/1431219165' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "793e383f-4d5a-41cb-b5b3-b4e4e07d6b8a"}]: dispatch 2021-11-20T06:01:39.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:39 smithi178 conmon[36600]: audit 2021-11-20T06:01:39.033187+0000 mon.smithi139 (mon.0) 317 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "793e383f-4d5a-41cb-b5b3-b4e4e07d6b8a"}]: dispatch 2021-11-20T06:01:39.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:39 smithi178 conmon[36600]: audit 2021-11-20T06:01:39.036855+0000 mon.smithi139 (mon.0) 318 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "793e383f-4d5a-41cb-b5b3-b4e4e07d6b8a"}]': finished 2021-11-20T06:01:39.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:39 smithi178 conmon[36600]: cluster 2021-11-20T06:01:39.036908+0000 mon.smithi139 (mon.0) 319 : cluster [DBG] osdmap e19: 4 total, 3 up, 4 in 2021-11-20T06:01:39.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:39 smithi178 conmon[36600]: audit 2021-11-20T06:01:39.037024+0000 mon.smithi139 (mon.0) 320 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:39.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: cluster 2021-11-20T06:01:37.169425+0000 osd.2 (osd.2) 1 : 2021-11-20T06:01:39.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: cluster [DBG] purged_snaps scrub starts 2021-11-20T06:01:39.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: cluster 2021-11-20T06:01:37.169635+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T06:01:39.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: cluster 2021 2021-11-20T06:01:39.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: -11-20T06:01:38.189863+0000 mgr.smithi139.dplyxn (mgr.14162) 112 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-20T06:01:39.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: audit 2021-11-20T06:01:38. 2021-11-20T06:01:39.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: 340927+0000 mon.smithi139 (mon.0) 312 : audit [DBG] 2021-11-20T06:01:39.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T06:01:39.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: cluster 2021-11-20T06:01:38 2021-11-20T06:01:39.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: .345724+0000 mon.smithi139 (mon.0) 313 : cluster 2021-11-20T06:01:39.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: [INF] osd.2 [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903] boot 2021-11-20T06:01:39.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: cluster 2021-11-20T06: 2021-11-20T06:01:39.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: 01:38.345755+0000 mon.smithi139 (mon.0) 2021-11-20T06:01:39.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: 314 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-11-20T06:01:39.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: audit 2021-11-20T 2021-11-20T06:01:39.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: 06:01:38.346011+0000 mon.smithi139 (mon. 2021-11-20T06:01:39.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: 0) 315 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-20T06:01:39.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: audit 2021-11 2021-11-20T06:01:39.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: -20T06:01:38.916180+0000 mon.smithi139 2021-11-20T06:01:39.786 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: (mon.0) 316 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:39.786 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: audit 2021-11-20T06:01:39.786 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: 2021-11-20T06:01:39.032786 2021-11-20T06:01:39.786 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: +0000 mon.smithi178 (mon.1) 4 : audit [INF] 2021-11-20T06:01:39.786 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: from='client.? 172.21.15.178:0/1431219165' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "793e383f-4d5a-41cb-b5b3-b4e4e07d6b8a"}]: dispatch 2021-11-20T06:01:39.787 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: audit 2021-11-20T06:01:39 2021-11-20T06:01:39.787 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: .033187+0000 mon.smithi139 (mon.0) 317 : audit 2021-11-20T06:01:39.787 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "793e383f-4d5a-41cb-b5b3-b4e4e07d6b8a"}]: dispatch 2021-11-20T06:01:39.787 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: audit 2021-11-20T06: 2021-11-20T06:01:39.787 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: 01:39.036855+0000 mon.smithi139 (mon.0) 2021-11-20T06:01:39.788 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: 318 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "793e383f-4d5a-41cb-b5b3-b4e4e07d6b8a"}]': finished 2021-11-20T06:01:39.788 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: cluster 2021-11-20T06:01: 2021-11-20T06:01:39.788 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: 39.036908+0000 mon.smithi139 (mon.0) 319 : 2021-11-20T06:01:39.788 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: cluster [DBG] osdmap e19: 4 total, 3 up, 4 in 2021-11-20T06:01:39.789 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: audit 2021-11-20T06:01:39.789 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: 2021-11-20T06:01:39 2021-11-20T06:01:39.789 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: .037024+0000 mon.smithi139 (mon.0) 320 : audit 2021-11-20T06:01:39.789 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:39 smithi139 conmon[33501]: [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:40.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:40 smithi178 conmon[36600]: audit 2021-11-20T06:01:39.432031+0000 mon.smithi178 (mon.1) 5 : audit [DBG] from='client.? 172.21.15.178:0/3225019630' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T06:01:40.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:40 smithi178 conmon[36600]: cluster 2021-11-20T06:01: 2021-11-20T06:01:40.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:40 smithi178 conmon[36600]: 40.040646+0000 mon.smithi139 (mon.0) 321 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2021-11-20T06:01:40.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:40 smithi178 conmon[36600]: audit 2021-11-20T06:01:40.040751+0000 mon.smithi139 (mon.0) 322 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:40.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:40 smithi139 conmon[33501]: audit 2021-11-20T06:01:39.432031+0000 mon.smithi178 (mon.1) 5 : 2021-11-20T06:01:40.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:40 smithi139 conmon[33501]: audit [DBG] from='client.? 172.21.15.178:0/3225019630' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T06:01:40.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:40 smithi139 conmon[33501]: cluster 2021-11-20T06:01:40.040646+0000 mon.smithi139 (mon.0) 321 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2021-11-20T06:01:40.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:40 smithi139 conmon[33501]: audit 2021-11-20T06:01:40.040751+0000 mon.smithi139 (mon.0) 322 : 2021-11-20T06:01:40.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:40 smithi139 conmon[33501]: audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:42.031 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:41 smithi139 conmon[33501]: cluster 2021-11-20T06:01:40.190211+0000 mgr.smithi139.dplyxn (mgr.14162) 113 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:42.031 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:41 smithi139 conmon[33501]: audit 2021-11-20T06:01:40.835804+0000 mon.smithi139 (mon.0) 323 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:42.178 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:41 smithi178 conmon[36600]: cluster 2021-11-20T06:01:40.190211+0000 mgr.smithi139.dplyxn (mgr.14162) 113 : 2021-11-20T06:01:42.178 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:41 smithi178 conmon[36600]: cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:42.179 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:41 smithi178 conmon[36600]: audit 2021-11-20T06:01:40.835804+0000 mon.smithi139 (mon.0) 323 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:44.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:44 smithi178 conmon[36600]: cluster 2021-11-20T06:01 2021-11-20T06:01:44.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:44 smithi178 conmon[36600]: :42.190489+0000 mgr.smithi139.dplyxn (mgr.14162) 114 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:44.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:44 smithi178 conmon[36600]: audit 2021-11-20T06:01:43.727453+0000 mon.smithi139 (mon.0) 324 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:44.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:44 smithi178 conmon[36600]: audit 2021-11-20T06:01:44.080692+0000 mon.smithi139 (mon.0) 325 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:44.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:44 smithi139 conmon[33501]: cluster 2021-11-20T06:01:42.190489+0000 mgr.smithi139.dplyxn (mgr.14162) 114 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:44.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:44 smithi139 conmon[33501]: audit 2021-11-20T06:01:43.727453+0000 mon.smithi139 (mon.0) 324 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:44.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:44 smithi139 conmon[33501]: audit 2021-11-20T06:01:44.080692+0000 mon.smithi139 (mon.0) 325 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:45.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:45 smithi178 conmon[36600]: audit 2021-11-20T06:01:45.052622+0000 mon.smithi139 (mon.0) 326 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-11-20T06:01:45.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:45 smithi178 conmon[36600]: audit 2021-11-20T06:01:45.053538+0000 mon.smithi139 (mon.0) 327 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:45.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:45 smithi139 conmon[33501]: audit 2021-11-20T06:01:45.052622+0000 mon.smithi139 (mon.0) 326 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-11-20T06:01:45.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:45 smithi139 conmon[33501]: audit 2021-11-20T06:01:45.053538+0000 mon.smithi139 (mon.0) 327 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:46.429 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:46 smithi178 conmon[36600]: cluster 2021-11-20T06:01:44.190781+0000 mgr.smithi139.dplyxn (mgr.14162) 115 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:46.429 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:46 smithi178 conmon[36600]: cephadm 2021-11-20T06:01:45.054279+0000 mgr.smithi139.dplyxn (mgr.14162) 116 : cephadm [INF] Deploying daemon osd.3 on smithi178 2021-11-20T06:01:46.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:46 smithi139 conmon[33501]: cluster 2021-11-20T06:01:44.190781+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:01:46.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:46 smithi139 conmon[33501]: mgr.14162) 115 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:46.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:46 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:45.054279+0000 2021-11-20T06:01:46.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:46 smithi139 conmon[33501]: mgr.smithi139.dplyxn (mgr.14162) 116 : cephadm [INF] Deploying daemon osd.3 on smithi178 2021-11-20T06:01:48.128 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 3 on host 'smithi178' 2021-11-20T06:01:48.129 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:48.139+0000 7f75967fc700 1 -- 172.21.15.178:0/632040460 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f75a0108d30 con 0x7f758c05c2a0 2021-11-20T06:01:48.129 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:48.142+0000 7f75a7dbe700 1 -- 172.21.15.178:0/632040460 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f758c05c2a0 msgr2=0x7f758c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:01:48.129 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:48.142+0000 7f75a7dbe700 1 --2- 172.21.15.178:0/632040460 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f758c05c2a0 0x7f758c05e760 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f759c005010 tx=0x7f759c009b30).stop 2021-11-20T06:01:48.130 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:48.142+0000 7f75a7dbe700 1 -- 172.21.15.178:0/632040460 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a00f6870 msgr2=0x7f75a0107460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:01:48.130 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:48.142+0000 7f75a7dbe700 1 --2- 172.21.15.178:0/632040460 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a00f6870 0x7f75a0107460 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f7590002700 tx=0x7f7590009e00).stop 2021-11-20T06:01:48.130 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:48.142+0000 7f75a7dbe700 1 -- 172.21.15.178:0/632040460 shutdown_connections 2021-11-20T06:01:48.130 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:48.142+0000 7f75a7dbe700 1 --2- 172.21.15.178:0/632040460 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f758c05c2a0 0x7f758c05e760 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:48.131 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:48.142+0000 7f75a7dbe700 1 --2- 172.21.15.178:0/632040460 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a00f55d0 0x7f75a0106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:48.131 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:48.142+0000 7f75a7dbe700 1 --2- 172.21.15.178:0/632040460 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a00f6870 0x7f75a0107460 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:48.131 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:48.142+0000 7f75a7dbe700 1 -- 172.21.15.178:0/632040460 >> 172.21.15.178:0/632040460 conn(0x7f75a00f0d70 msgr2=0x7f75a00fa2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:01:48.131 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:48.142+0000 7f75a7dbe700 1 -- 172.21.15.178:0/632040460 shutdown_connections 2021-11-20T06:01:48.132 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:48.142+0000 7f75a7dbe700 1 -- 172.21.15.178:0/632040460 wait complete. 2021-11-20T06:01:48.494 DEBUG:teuthology.orchestra.run.smithi178:osd.3> sudo journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.3.service 2021-11-20T06:01:48.497 INFO:tasks.cephadm:Deploying osd.4 on smithi178 with /dev/vg_nvme/lv_3... 2021-11-20T06:01:48.498 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-11-20T06:01:48.525 INFO:journalctl@ceph.osd.3.smithi178.stdout:-- Logs begin at Sat 2021-11-20 05:48:12 UTC. -- 2021-11-20T06:01:48.692 INFO:journalctl@ceph.osd.3.smithi178.stdout:Nov 20 06:01:48 smithi178 conmon[43163]: debug 2021-11-20T06:01:48.578+0000 7fc61735f080 -1 Falling back to public interface 2021-11-20T06:01:48.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:48 smithi178 conmon[36600]: cluster 2021-11-20T06:01:46.191119+0000 mgr.smithi139.dplyxn (mgr.14162) 117 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:48.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:48 smithi178 conmon[36600]: audit 2021-11-20T06:01:48.129125+0000 mon.smithi139 (mon.0) 328 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:01:48.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:48 smithi178 conmon[36600]: audit 2021-11-20T06:01:48.133852+0000 mon.smithi139 (mon.0) 329 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:48.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:48 smithi178 conmon[36600]: audit 2021-11-20T06:01:48.140311+0000 mon.smithi139 (mon.0) 330 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:48.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:48 smithi139 conmon[33501]: cluster 2021-11-20T06:01:46.191119+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:01:48.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:48 smithi139 conmon[33501]: 14162) 117 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:48.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:48 smithi139 conmon[33501]: audit 2021-11-20T06:01:48.129125+0000 mon.smithi139 (mon.0) 328 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:01:48.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:48 smithi139 conmon[33501]: 2021-11-20T06:01:48.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:48 smithi139 conmon[33501]: audit 2021-11-20T06: 2021-11-20T06:01:48.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:48 smithi139 conmon[33501]: 01:48.133852+0000 mon.smithi139 (mon.0) 329 : audit [INF] 2021-11-20T06:01:48.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:48 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:48.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:48 smithi139 conmon[33501]: audit 2021-11-20T06: 2021-11-20T06:01:48.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:48 smithi139 conmon[33501]: 01:48.140311+0000 mon.smithi139 (mon.0) 330 2021-11-20T06:01:48.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:48 smithi139 conmon[33501]: : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:49.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:49 smithi178 conmon[36600]: audit 2021-11-20T06:01:48.142986+0000 mon.smithi139 (mon. 2021-11-20T06:01:49.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:49 smithi178 conmon[36600]: 0) 331 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-11-20T06:01:49.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:49 smithi178 conmon[36600]: cluster 2021-11-20T06:01:48.191444+0000 mgr.smithi139.dplyxn (mgr.14162) 118 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:49.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:49 smithi178 conmon[36600]: audit 2021-11-20T06:01:49.145260+0000 mon.smithi139 (mon.0) 332 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-11-20T06:01:49.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:49 smithi178 conmon[36600]: cluster 2021-11-20T06:01:49.145336+0000 mon.smithi139 (mon.0) 333 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2021-11-20T06:01:49.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:49 smithi178 conmon[36600]: audit 2021-11-20T06:01:49.145483+0000 mon.smithi139 (mon.0) 334 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:49.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:49 smithi178 conmon[36600]: audit 2021-11-20T06:01:49.146200+0000 mon.smithi139 (mon.0) 335 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-11-20T06:01:49.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:49 smithi139 conmon[33501]: audit 2021-11-20T06:01:48.142986+0000 mon.smithi139 ( 2021-11-20T06:01:49.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:49 smithi139 conmon[33501]: mon.0) 331 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-11-20T06:01:49.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:49 smithi139 conmon[33501]: cluster 2021-11-20T06:01:48.191444+0000 mgr.smithi139.dplyxn (mgr.14162) 118 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:49.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:49 smithi139 conmon[33501]: audit 2021-11-20T06:01:49.145260+0000 mon.smithi139 (mon.0) 332 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-11-20T06:01:49.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:49 smithi139 conmon[33501]: cluster 2021-11-20T06:01:49.145336+0000 mon.smithi139 (mon.0) 333 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2021-11-20T06:01:49.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:49 smithi139 conmon[33501]: audit 2021-11-20T06:01:49.145483+0000 mon.smithi139 (mon.0) 334 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:49.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:49 smithi139 conmon[33501]: audit 2021-11-20T06:01:49.146200+0000 mon.smithi139 (mon.0) 335 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-11-20T06:01:50.529 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-11-20T06:01:50.529 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-11-20T06:01:50.530 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2021-11-20T06:01:50.530 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2021-11-20T06:01:50.530 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00925157 s, 1.1 GB/s 2021-11-20T06:01:50.530 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2021-11-20T06:01:50.937 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi178:vg_nvme/lv_3 2021-11-20T06:01:51.192 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:51 smithi178 conmon[36600]: audit 2021-11-20T06:01:50.147124+0000 mon.smithi139 (mon.0) 336 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-11-20T06:01:51.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:51 smithi178 conmon[36600]: cluster 2021-11-20T06:01:50.147213+0000 mon.smithi139 (mon.0) 337 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-11-20T06:01:51.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:51 smithi178 conmon[36600]: audit 2021-11-20T06:01:50.147353+0000 mon.smithi139 (mon.0) 338 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:51.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:51 smithi139 conmon[33501]: audit 2021-11-20T06:01:50.147124+0000 mon.smithi139 (mon.0) 336 : audit 2021-11-20T06:01:51.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:51 smithi139 conmon[33501]: [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-11-20T06:01:51.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:51 smithi139 conmon[33501]: cluster 2021-11-20T06:01:50.147213+0000 mon.smithi139 (mon.0) 337 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-11-20T06:01:51.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:51 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:01:51.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:51 smithi139 conmon[33501]: 20T06:01:50.147353+0000 mon.smithi139 2021-11-20T06:01:51.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:51 smithi139 conmon[33501]: (mon.0) 338 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:51.926 INFO:journalctl@ceph.osd.3.smithi178.stdout:Nov 20 06:01:51 smithi178 conmon[43163]: debug 2021-11-20T06:01:51.831+0000 7fc61735f080 -1 osd.3 0 log_to_monitors {default=true} 2021-11-20T06:01:52.177 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:52 smithi178 conmon[36600]: cluster 2021-11-20T06:01:50.191756+0000 mgr.smithi139.dplyxn (mgr 2021-11-20T06:01:52.178 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:52 smithi178 conmon[36600]: .14162) 119 : cluster [DBG] pgmap v83: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:52.178 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:52 smithi178 conmon[36600]: cluster 2021-11-20T06:01:51.153792+0000 mon.smithi139 (mon.0) 339 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-11-20T06:01:52.178 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:52 smithi178 conmon[36600]: audit 2021-11-20T06:01:51.153919+0000 mon.smithi139 (mon.0) 340 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:52.179 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:52 smithi178 conmon[36600]: audit 2021-11-20T06:01:51.336607+0000 mon.smithi139 (mon.0) 341 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:52.179 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:52 smithi178 conmon[36600]: audit 2021-11-20T06:01:51.835295+0000 mon.smithi178 (mon.1) 6 : audit [INF] from='osd.3 [v2:172.21.15.178:6800/927121970,v1:172.21.15.178:6801/927121970]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T06:01:52.179 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:52 smithi178 conmon[36600]: audit 2021-11-20T06:01:51.835579+0000 mon.smithi139 (mon.0) 342 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T06:01:52.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: cluster 2021-11-20T06:01:50.191756+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:01:52.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: ) 119 : cluster [DBG] pgmap v83: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:52.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: cluster 2021-11-20T06:01:51.153792+0000 mon.smithi139 (mon.0) 339 : cluster [DBG] 2021-11-20T06:01:52.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: osdmap e23: 4 total, 3 up, 4 in 2021-11-20T06:01:52.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: audit 2021 2021-11-20T06:01:52.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: -11-20T06:01:51.153919+0000 mon.smithi139 2021-11-20T06:01:52.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: (mon.0) 340 : audit [DBG] 2021-11-20T06:01:52.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:52.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: audit 2021 2021-11-20T06:01:52.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: -11-20T06:01:51 2021-11-20T06:01:52.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: .336607+0000 mon.smithi139 (mon.0) 2021-11-20T06:01:52.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: 341 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:52.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: audit 2021- 2021-11-20T06:01:52.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: 11-20T06:01:51.835295 2021-11-20T06:01:52.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: +0000 mon.smithi178 (mon.1) 6 2021-11-20T06:01:52.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: : audit [INF] from='osd.3 [v2:172.21.15.178:6800/927121970,v1:172.21.15.178:6801/927121970]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T06:01:52.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: audit 2021-11-20 2021-11-20T06:01:52.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: T06:01:51.835579+0000 mon.smithi139 2021-11-20T06:01:52.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: (mon.0) 342 : audit [INF] 2021-11-20T06:01:52.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:52 smithi139 conmon[33501]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-20T06:01:52.959 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.970+0000 7f5938ab7700 1 -- 172.21.15.178:0/4134857450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59340f55d0 msgr2=0x7f59340f59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:01:52.959 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.970+0000 7f5938ab7700 1 --2- 172.21.15.178:0/4134857450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59340f55d0 0x7f59340f59f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f5924004660 tx=0x7f5924009b30).stop 2021-11-20T06:01:52.959 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.970+0000 7f5938ab7700 1 -- 172.21.15.178:0/4134857450 shutdown_connections 2021-11-20T06:01:52.960 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.970+0000 7f5938ab7700 1 --2- 172.21.15.178:0/4134857450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59340f6870 0x7f59340f6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:52.960 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.970+0000 7f5938ab7700 1 --2- 172.21.15.178:0/4134857450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59340f55d0 0x7f59340f59f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:52.960 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.970+0000 7f5938ab7700 1 -- 172.21.15.178:0/4134857450 >> 172.21.15.178:0/4134857450 conn(0x7f59340f0d70 msgr2=0x7f59340f3190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:01:52.960 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.970+0000 7f5938ab7700 1 -- 172.21.15.178:0/4134857450 shutdown_connections 2021-11-20T06:01:52.960 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.970+0000 7f5938ab7700 1 -- 172.21.15.178:0/4134857450 wait complete. 2021-11-20T06:01:52.961 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.970+0000 7f5938ab7700 1 Processor -- start 2021-11-20T06:01:52.961 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f5938ab7700 1 -- start start 2021-11-20T06:01:52.961 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f5938ab7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59340f55d0 0x7f5934105eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:01:52.962 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f5938ab7700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59340f6870 0x7f59341063a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:01:52.962 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f5938ab7700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5934106950 con 0x7f59340f55d0 2021-11-20T06:01:52.962 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f5938ab7700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5934106a90 con 0x7f59340f6870 2021-11-20T06:01:52.962 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f5931d9b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59340f6870 0x7f59341063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:01:52.962 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f5931d9b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59340f6870 0x7f59341063a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:50162/0 (socket says 172.21.15.178:50162) 2021-11-20T06:01:52.963 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f5931d9b700 1 -- 172.21.15.178:0/1439929942 learned_addr learned my addr 172.21.15.178:0/1439929942 (peer_addr_for_me v2:172.21.15.178:0/0) 2021-11-20T06:01:52.963 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f593259c700 1 --2- 172.21.15.178:0/1439929942 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59340f55d0 0x7f5934105eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:01:52.963 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f5931d9b700 1 -- 172.21.15.178:0/1439929942 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59340f55d0 msgr2=0x7f5934105eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:01:52.963 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f5931d9b700 1 --2- 172.21.15.178:0/1439929942 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59340f55d0 0x7f5934105eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:01:52.964 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f5931d9b700 1 -- 172.21.15.178:0/1439929942 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5924005040 con 0x7f59340f6870 2021-11-20T06:01:52.964 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f5931d9b700 1 --2- 172.21.15.178:0/1439929942 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59340f6870 0x7f59341063a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f592800f940 tx=0x7f5928009de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:01:52.964 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f5922ffd700 1 -- 172.21.15.178:0/1439929942 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5928009500 con 0x7f59340f6870 2021-11-20T06:01:52.964 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f5922ffd700 1 -- 172.21.15.178:0/1439929942 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5928009660 con 0x7f59340f6870 2021-11-20T06:01:52.965 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.971+0000 7f5938ab7700 1 -- 172.21.15.178:0/1439929942 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5934107430 con 0x7f59340f6870 2021-11-20T06:01:52.965 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.972+0000 7f5938ab7700 1 -- 172.21.15.178:0/1439929942 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59341079f0 con 0x7f59340f6870 2021-11-20T06:01:52.965 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.972+0000 7f5922ffd700 1 -- 172.21.15.178:0/1439929942 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f592800de30 con 0x7f59340f6870 2021-11-20T06:01:52.966 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.972+0000 7f5938ab7700 1 -- 172.21.15.178:0/1439929942 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5934045ad0 con 0x7f59340f6870 2021-11-20T06:01:52.966 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.973+0000 7f5922ffd700 1 -- 172.21.15.178:0/1439929942 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f592800e040 con 0x7f59340f6870 2021-11-20T06:01:52.966 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.973+0000 7f5922ffd700 1 --2- 172.21.15.178:0/1439929942 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f591c05c2b0 0x7f591c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:01:52.966 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.973+0000 7f5922ffd700 1 -- 172.21.15.178:0/1439929942 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 3282+0+0 (secure 0 0 0) 0x7f5928077ff0 con 0x7f59340f6870 2021-11-20T06:01:52.967 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.973+0000 7f593259c700 1 --2- 172.21.15.178:0/1439929942 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f591c05c2b0 0x7f591c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:01:52.967 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.974+0000 7f593259c700 1 --2- 172.21.15.178:0/1439929942 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f591c05c2b0 0x7f591c05e770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f5924009f50 tx=0x7f5924009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:01:52.968 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:52.977+0000 7f5922ffd700 1 -- 172.21.15.178:0/1439929942 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f592804b560 con 0x7f59340f6870 2021-11-20T06:01:53.149 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:01:53.163+0000 7f5938ab7700 1 -- 172.21.15.178:0/1439929942 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f59340ff830 con 0x7f591c05c2b0 2021-11-20T06:01:53.639 INFO:journalctl@ceph.osd.3.smithi178.stdout:Nov 20 06:01:53 smithi178 conmon[43163]: debug 2021-11-20T06:01:53.348+0000 7fc60d8d0700 -1 osd.3 0 waiting for initial osdmap 2021-11-20T06:01:53.639 INFO:journalctl@ceph.osd.3.smithi178.stdout:Nov 20 06:01:53 smithi178 conmon[43163]: debug 2021-11-20T06:01:53.357+0000 7fc608849700 -1 osd.3 25 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T06:01:53.640 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:53 smithi178 conmon[36600]: cluster 2021-11-20T06:01:52.192008+0000 mgr.smithi139.dplyxn (mgr.14162) 120 : cluster [DBG] pgmap v85: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:53.640 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:53 smithi178 conmon[36600]: audit 2021-11-20T06:01:52.342713+0000 mon.smithi139 (mon.0) 343 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-11-20T06:01:53.640 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:53 smithi178 conmon[36600]: cluster 2021-11-20T06:01:52.342806+0000 mon.smithi139 (mon.0) 344 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-11-20T06:01:53.640 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:53 smithi178 conmon[36600]: audit 2021-11-20T06:01:52.343073+0000 mon.smithi139 (mon.0) 345 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:53.641 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:53 smithi178 conmon[36600]: audit 2021-11-20T06:01:52.343502+0000 mon.smithi178 (mon.1) 7 : audit [INF] from='osd.3 [v2:172.21.15.178:6800/927121970,v1:172.21.15.178:6801/927121970]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2021-11-20T06:01:53.641 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:53 smithi178 conmon[36600]: audit 2021-11-20T06:01:52.343799+0000 mon.smithi139 (mon.0) 346 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2021-11-20T06:01:53.641 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:53 smithi178 conmon[36600]: audit 2021-11-20T06:01:52.750919+0000 mon.smithi139 (mon.0) 347 : audit [INF] 2021-11-20T06:01:53.641 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:53 smithi178 conmon[36600]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:53.642 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:53 smithi178 conmon[36600]: audit 2021-11-20T06:01:53.164903+0000 mgr.smithi139.dplyxn (mgr.14162) 121 : audit [DBG] from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:01:53.642 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:53 smithi178 conmon[36600]: audit 2021-11-20T06:01:53.165589+0000 mon.smithi139 (mon.0) 348 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T06:01:53.642 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:53 smithi178 conmon[36600]: cephadm 2021-11-20T06:01:53.166587+0000 mgr.smithi139.dplyxn (mgr.14162) 122 : cephadm [INF] Applying service osd.None on host smithi178... 2021-11-20T06:01:53.642 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:53 smithi178 conmon[36600]: audit 2021-11-20T06:01:53.166785+0000 mon.smithi139 (mon.0) 349 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T06:01:53.642 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:53 smithi178 conmon[36600]: audit 2021-11-20T06:01:53.167381+0000 mon.smithi139 (mon.0) 350 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:53.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: cluster 2021-11-20T06:01:52.192008+0000 mgr.smithi139.dplyxn (mgr.14162) 120 : cluster [DBG] pgmap v85: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:53.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: audit 2021-11-20T06:01:52.342713+0000 mon.smithi139 (mon.0) 343 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-11-20T06:01:53.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: cluster 2021-11-20T06:01:52.342806+0000 mon.smithi139 (mon.0) 344 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-11-20T06:01:53.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: audit 2021-11-20T06:01:52.343073+0000 mon.smithi139 (mon.0) 345 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:53.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: audit 2021-11-20T06:01:53.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: 2021-11-20T06:01:52.343502+0000 mon.smithi178 (mon.1) 7 : audit [INF] from='osd.3 [v2:172.21.15.178:6800/927121970,v1:172.21.15.178:6801/927121970]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2021-11-20T06:01:53.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: audit 2021-11-20T06:01:53.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: 2021-11-20T06: 2021-11-20T06:01:53.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: 01:52.343799+0000 mon.smithi139 (mon.0) 346 : audit [INF] 2021-11-20T06:01:53.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2021-11-20T06:01:53.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: audit 2021-11-20T06:01:53.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: 2021-11- 2021-11-20T06:01:53.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: 20T06:01:52.750919+0000 mon.smithi139 (mon.0 2021-11-20T06:01:53.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: ) 347 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:53.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: audit 2021-11-20T06:01:53.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: 2021- 2021-11-20T06:01:53.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: 11-20T06:01:53.164903+0000 mgr.smithi139.dplyxn 2021-11-20T06:01:53.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: (mgr.14162) 121 : audit [DBG] from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:01:53.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: audit 2021-11-20T06:01:53.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: 2021-11-20T06:01:53.165589+0000 2021-11-20T06:01:53.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: mon.smithi139 (mon.0) 348 : audit [DBG] 2021-11-20T06:01:53.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T06:01:53.783 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:53.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: 2021-11-20T06:01:53.166587+0000 2021-11-20T06:01:53.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: mgr.smithi139.dplyxn (mgr.14162) 122 : cephadm [INF] 2021-11-20T06:01:53.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: Applying service osd.None on host smithi178... 2021-11-20T06:01:53.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: audit 2021- 2021-11-20T06:01:53.784 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: 11-20T06:01:53.166785+0000 mon.smithi139 ( 2021-11-20T06:01:53.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: mon.0) 349 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T06:01:53.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: audit 2021-11-20T06:01:53.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: 2021-11-20T06:01:53.167381+0000 2021-11-20T06:01:53.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: mon.smithi139 (mon.0) 350 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:01:53.785 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:53 smithi139 conmon[33501]: 2021-11-20T06:01:54.624 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:54 smithi178 conmon[36600]: audit 2021-11-20T06:01:53.346001+0000 mon.smithi139 (mon.0) 351 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2021-11-20T06:01:54.624 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:54 smithi178 conmon[36600]: cluster 2021-11-20T06:01:53.346066+0000 mon.smithi139 (mon.0) 352 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2021-11-20T06:01:54.624 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:54 smithi178 conmon[36600]: audit 2021-11-20T06:01:53.347060+0000 mon.smithi139 (mon.0) 353 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:54.625 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:54 smithi178 conmon[36600]: audit 2021-11-20T06:01:53.350695+0000 mon.smithi139 (mon.0) 354 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:54.625 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:54 smithi178 conmon[36600]: cluster 2021-11-20T06:01:54.345996+0000 mon.smithi139 (mon.0) 355 : cluster [INF] osd.3 [v2:172.21.15.178:6800/927121970,v1:172.21.15.178:6801/927121970] boot 2021-11-20T06:01:54.625 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:54 smithi178 conmon[36600]: cluster 2021-11-20T06:01:54.346049+0000 mon.smithi139 (mon.0) 356 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-11-20T06:01:54.625 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:54 smithi178 conmon[36600]: audit 2021-11-20T06:01:54.346196+0000 mon.smithi139 (mon.0) 357 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:54.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:54 smithi139 conmon[33501]: audit 2021-11-20T06:01:53.346001+0000 mon.smithi139 (mon.0) 351 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2021-11-20T06:01:54.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:54 smithi139 conmon[33501]: cluster 2021-11-20T06:01:53.346066+0000 mon.smithi139 2021-11-20T06:01:54.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:54 smithi139 conmon[33501]: (mon.0) 352 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2021-11-20T06:01:54.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:54 smithi139 conmon[33501]: audit 2021-11-20T06:01:53.347060+0000 mon.smithi139 (mon.0) 353 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:54.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:54 smithi139 conmon[33501]: audit 2021-11-20T06:01:53.350695+0000 mon.smithi139 (mon.0) 354 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:54.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:54 smithi139 conmon[33501]: cluster 2021-11-20T06:01:54.345996+0000 mon.smithi139 (mon.0) 355 : cluster [INF] osd.3 [v2:172.21.15.178:6800/927121970,v1:172.21.15.178:6801/927121970] boot 2021-11-20T06:01:54.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:54 smithi139 conmon[33501]: cluster 2021-11-20T06: 2021-11-20T06:01:54.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:54 smithi139 conmon[33501]: 01:54.346049+0000 mon.smithi139 (mon.0) 356 2021-11-20T06:01:54.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:54 smithi139 conmon[33501]: : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-11-20T06:01:54.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:54 smithi139 conmon[33501]: audit 2021-11-20T06 2021-11-20T06:01:54.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:54 smithi139 conmon[33501]: :01:54.346196+0000 mon.smithi139 (mon.0 2021-11-20T06:01:54.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:54 smithi139 conmon[33501]: ) 357 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-20T06:01:55.942 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:55 smithi178 conmon[36600]: cluster 2021-11-20T06:01:52.838128+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T06:01:55.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:55 smithi178 conmon[36600]: cluster 2021-11-20T06:01:52.838239+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T06:01:55.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:55 smithi178 conmon[36600]: cluster 2021-11-20T06:01:54.192440+0000 mgr.smithi139.dplyxn (mgr.14162) 123 : cluster [DBG] pgmap v88: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:55.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:55 smithi178 conmon[36600]: cephadm 2021-11-20T06:01:54.643792+0000 mgr.smithi139.dplyxn (mgr.14162) 124 : cephadm [INF] Detected new or changed devices on smithi178 2021-11-20T06:01:55.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:55 smithi178 conmon[36600]: audit 2021-11-20T06:01:54.648656+0000 mon.smithi139 (mon.0) 358 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:55.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:55 smithi178 conmon[36600]: audit 2021-11-20T06:01:54.652329+0000 mon.smithi139 (mon.0) 359 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:55.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:55 smithi178 conmon[36600]: audit 2021-11-20T06:01:54.655325+0000 mon.smithi139 (mon.0) 360 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:01:55.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:55 smithi178 conmon[36600]: cluster 2021-11-20T06:01:55.346590+0000 mon.smithi139 (mon.0) 361 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2021-11-20T06:01:55.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:55 smithi178 conmon[36600]: audit 2021-11-20T06:01:55.371149+0000 mon.smithi178 (mon.1) 8 : audit [INF] from='client.? 172.21.15.178:0/2652805743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e453fbf-8ef9-4227-b3b8-cd6631bff6ab"}]: dispatch 2021-11-20T06:01:55.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:55 smithi178 conmon[36600]: audit 2021-11-20T06:01:55.371501+0000 mon.smithi139 (mon.0) 362 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e453fbf-8ef9-4227-b3b8-cd6631bff6ab"}]: dispatch 2021-11-20T06:01:55.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:55 smithi178 conmon[36600]: audit 2021-11-20T06:01:55.377555+0000 mon.smithi139 (mon.0) 363 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e453fbf-8ef9-4227-b3b8-cd6631bff6ab"}]': finished 2021-11-20T06:01:55.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:55 smithi178 conmon[36600]: cluster 2021-11-20T06:01:55.377609+0000 mon.smithi139 (mon.0) 364 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-11-20T06:01:55.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:55 smithi178 conmon[36600]: audit 2021-11-20T06:01:55.377909+0000 mon.smithi139 (mon.0) 365 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T06:01:56.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: cluster 2021-11-20T06:01:52.838128+0000 osd.3 (osd.3) 1 2021-11-20T06:01:56.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: : cluster [DBG] purged_snaps scrub starts 2021-11-20T06:01:56.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: cluster 2021-11-20T06:01:52.838239+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T06:01:56.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: cluster 2021-11- 2021-11-20T06:01:56.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: 20T06:01:54.192440+0000 mgr.smithi139.dplyxn (mgr.14162) 123 : cluster [DBG] pgmap v88: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-20T06:01:56.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: cephadm 2021-11-20T06:01:54 2021-11-20T06:01:56.029 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: .643792+0000 mgr.smithi139.dplyxn (mgr.14162) 124 : cephadm [INF] Detected new or changed devices on smithi178 2021-11-20T06:01:56.030 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: audit 2021-11-20T06:01:54.648656+0000 mon.smithi139 ( 2021-11-20T06:01:56.030 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: mon.0) 358 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:56.030 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: audit 2021-11-20T06:01:54. 2021-11-20T06:01:56.030 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: 652329+0000 mon.smithi139 (mon.0) 359 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:01:56.031 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:01:56.031 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: 20T06:01:54.655325+0000 mon.smithi139 (mon.0) 360 : audit [DBG] 2021-11-20T06:01:56.031 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:01:56.031 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: cluster 2021-11 2021-11-20T06:01:56.031 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: -20T06:01:55.346590+0000 mon.smithi139 (mon.0) 361 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2021-11-20T06:01:56.032 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:01:56.032 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: 20T06:01:55.371149+0000 mon.smithi178 (mon.1) 8 : 2021-11-20T06:01:56.032 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: audit [INF] from='client.? 172.21.15.178:0/2652805743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e453fbf-8ef9-4227-b3b8-cd6631bff6ab"}]: dispatch 2021-11-20T06:01:56.032 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: audit 2021-11-20T06:01:55.371501+0000 mon.smithi139 2021-11-20T06:01:56.033 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: (mon.0) 362 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e453fbf-8ef9-4227-b3b8-cd6631bff6ab"}]: dispatch 2021-11-20T06:01:56.033 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: audit 2021-11-20T06 2021-11-20T06:01:56.033 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: :01:55.377555+0000 mon.smithi139 (mon.0) 363 : audit [INF] 2021-11-20T06:01:56.033 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e453fbf-8ef9-4227-b3b8-cd6631bff6ab"}]': finished 2021-11-20T06:01:56.034 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: cluster 2021-11-20T06:01:56.034 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: 2021-11-20T06:01:55.377609+0000 mon.smithi139 (mon.0) 364 : cluster 2021-11-20T06:01:56.035 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-11-20T06:01:56.035 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: audit 2021-11-20T06:01:55.377909+0000 mon.smithi139 ( 2021-11-20T06:01:56.035 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:55 smithi139 conmon[33501]: mon.0) 365 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T06:01:56.942 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:56 smithi178 conmon[36600]: audit 2021-11-20T06:01:55.779581+0000 mon.smithi178 (mon.1) 9 : audit [DBG] 2021-11-20T06:01:56.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:56 smithi178 conmon[36600]: from='client.? 172.21.15.178:0/3597686978' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T06:01:57.027 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:56 smithi139 conmon[33501]: audit 2021-11-20T06:01:55.779581+0000 mon.smithi178 (mon.1) 9 : audit 2021-11-20T06:01:57.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:56 smithi139 conmon[33501]: [DBG] from='client.? 172.21.15.178:0/3597686978' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T06:01:58.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:58 smithi178 conmon[36600]: cluster 2021-11-20T06:01:56.192751+0000 mgr.smithi139.dplyxn (mgr.14162) 125 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:01:58.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:01:58 smithi178 conmon[36600]: 2021-11-20T06:01:58.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:58 smithi139 conmon[33501]: cluster 2021-11-20T06:01:56.192751+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:01:58.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:01:58 smithi139 conmon[33501]: ) 125 : cluster [DBG] pgmap v92: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:02:00.400 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:00 smithi178 conmon[36600]: cluster 2021-11-20T06:01:58.193153+0000 mgr.smithi139.dplyxn (mgr.14162) 126 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:02:00.400 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:00 smithi178 conmon[36600]: audit 2021-11-20T06:01:59.256461+0000 mon.smithi139 (mon.0) 366 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:00.401 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:00 smithi178 conmon[36600]: audit 2021-11-20T06:01:59.259349+0000 mon.smithi139 (mon.0) 367 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:00.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:00 smithi139 conmon[33501]: cluster 2021-11-20T06:01:58.193153+0000 mgr.smithi139.dplyxn (mgr.14162) 126 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:02:00.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:00 smithi139 conmon[33501]: audit 2021-11-20T06:01:59.256461+0000 mon.smithi139 (mon.0) 366 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:00.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:00 smithi139 conmon[33501]: audit 2021-11-20T06:01:59.259349+0000 mon.smithi139 (mon.0) 367 : audit [INF] 2021-11-20T06:02:00.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:00 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:01.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:01 smithi178 conmon[36600]: audit 2021-11-20T06: 2021-11-20T06:02:01.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:01 smithi178 conmon[36600]: 02:01.151521+0000 mon.smithi139 (mon.0) 368 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-11-20T06:02:01.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:01 smithi178 conmon[36600]: audit 2021-11-20T06:02:01.152333+0000 mon.smithi139 (mon.0) 369 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:02:01.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:01 smithi139 conmon[33501]: audit 2021-11-20T06:02:01.151521+0000 mon.smithi139 (mon 2021-11-20T06:02:01.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:01 smithi139 conmon[33501]: .0) 368 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-11-20T06:02:01.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:01 smithi139 conmon[33501]: audit 2021-11-20T06:02:01.152333+0000 mon.smithi139 (mon.0) 369 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:02:02.299 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:02 smithi178 conmon[36600]: cluster 2021-11-20T06:02:00.193550+0000 mgr.smithi139.dplyxn (mgr.14162) 127 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:02:02.300 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:02 smithi178 conmon[36600]: cephadm 2021-11-20T06:02:01.153013+0000 mgr.smithi139.dplyxn (mgr.14162) 128 : cephadm [INF] Deploying daemon osd.4 on smithi178 2021-11-20T06:02:02.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:02 smithi139 conmon[33501]: cluster 2021-11-20T06:02:00.193550+0000 mgr.smithi139.dplyxn (mgr.14162) 127 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:02:02.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:02 smithi139 conmon[33501]: cephadm 2021-11-20T06:02:01.153013+0000 mgr.smithi139.dplyxn (mgr.14162) 128 : cephadm [INF] Deploying daemon osd.4 on smithi178 2021-11-20T06:02:03.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:03 smithi178 conmon[36600]: cluster 2021-11-20T06:02:02.193858+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:02:03.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:03 smithi178 conmon[36600]: mgr.14162) 129 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:02:03.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:03 smithi139 conmon[33501]: cluster 2021-11-20T06:02:02.193858+0000 mgr.smithi139.dplyxn (mgr.14162) 129 : cluster 2021-11-20T06:02:03.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:03 smithi139 conmon[33501]: [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:02:04.126 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:04.138+0000 7f5922ffd700 1 -- 172.21.15.178:0/1439929942 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f59340ff830 con 0x7f591c05c2b0 2021-11-20T06:02:04.126 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 4 on host 'smithi178' 2021-11-20T06:02:04.127 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:04.140+0000 7f5938ab7700 1 -- 172.21.15.178:0/1439929942 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f591c05c2b0 msgr2=0x7f591c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:04.127 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:04.140+0000 7f5938ab7700 1 --2- 172.21.15.178:0/1439929942 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f591c05c2b0 0x7f591c05e770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f5924009f50 tx=0x7f5924009b30).stop 2021-11-20T06:02:04.128 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:04.140+0000 7f5938ab7700 1 -- 172.21.15.178:0/1439929942 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59340f6870 msgr2=0x7f59341063a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:04.128 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:04.140+0000 7f5938ab7700 1 --2- 172.21.15.178:0/1439929942 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59340f6870 0x7f59341063a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f592800f940 tx=0x7f5928009de0).stop 2021-11-20T06:02:04.129 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:04.140+0000 7f5938ab7700 1 -- 172.21.15.178:0/1439929942 shutdown_connections 2021-11-20T06:02:04.129 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:04.140+0000 7f5938ab7700 1 --2- 172.21.15.178:0/1439929942 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f591c05c2b0 0x7f591c05e770 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:04.129 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:04.140+0000 7f5938ab7700 1 --2- 172.21.15.178:0/1439929942 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59340f55d0 0x7f5934105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:04.130 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:04.140+0000 7f5938ab7700 1 --2- 172.21.15.178:0/1439929942 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59340f6870 0x7f59341063a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:04.130 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:04.140+0000 7f5938ab7700 1 -- 172.21.15.178:0/1439929942 >> 172.21.15.178:0/1439929942 conn(0x7f59340f0d70 msgr2=0x7f59340fa2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:04.130 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:04.140+0000 7f5938ab7700 1 -- 172.21.15.178:0/1439929942 shutdown_connections 2021-11-20T06:02:04.130 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:04.140+0000 7f5938ab7700 1 -- 172.21.15.178:0/1439929942 wait complete. 2021-11-20T06:02:04.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:04 smithi178 conmon[36600]: audit 2021-11-20T06:02:04 2021-11-20T06:02:04.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:04 smithi178 conmon[36600]: .129026+0000 mon.smithi139 (mon.0) 370 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:02:04.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:04 smithi178 conmon[36600]: audit 2021-11-20T06:02: 2021-11-20T06:02:04.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:04 smithi178 conmon[36600]: 04.133449+0000 mon.smithi139 (mon.0) 2021-11-20T06:02:04.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:04 smithi178 conmon[36600]: 371 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:04.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:04 smithi178 conmon[36600]: audit 2021 2021-11-20T06:02:04.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:04 smithi178 conmon[36600]: -11-20T06:02:04. 2021-11-20T06:02:04.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:04 smithi178 conmon[36600]: 138227+0000 mon.smithi139 (mon.0) 2021-11-20T06:02:04.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:04 smithi178 conmon[36600]: 372 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:04.525 DEBUG:teuthology.orchestra.run.smithi178:osd.4> sudo journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.4.service 2021-11-20T06:02:04.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:04 smithi139 conmon[33501]: audit 2021-11-20T06:02:04.129026+0000 mon.smithi139 (mon.0) 370 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:02:04.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:04 smithi139 conmon[33501]: audit 2021-11-20T06:02:04.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:04 smithi139 conmon[33501]: 2021-11-20T06:02:04.133449+0000 mon.smithi139 (mon.0) 371 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:04.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:04 smithi139 conmon[33501]: audit 2021-11-20T06:02:04.138227+0000 mon.smithi139 (mon.0) 372 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:04.529 INFO:tasks.cephadm:Deploying osd.5 on smithi178 with /dev/vg_nvme/lv_2... 2021-11-20T06:02:04.529 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-11-20T06:02:04.567 INFO:journalctl@ceph.osd.4.smithi178.stdout:-- Logs begin at Sat 2021-11-20 05:48:12 UTC. -- 2021-11-20T06:02:05.248 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:05 smithi178 conmon[36600]: cluster 2021-11-20T06:02:04.194222+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:02:05.248 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:05 smithi178 conmon[36600]: ) 130 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:02:05.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:05 smithi139 conmon[33501]: cluster 2021-11-20T06:02:04.194222+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:02:05.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:05 smithi139 conmon[33501]: ) 130 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:02:06.482 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-11-20T06:02:06.483 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-11-20T06:02:06.483 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2021-11-20T06:02:06.483 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2021-11-20T06:02:06.483 INFO:teuthology.orchestra.run.smithi178.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00927749 s, 1.1 GB/s 2021-11-20T06:02:06.484 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2021-11-20T06:02:06.975 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi178:vg_nvme/lv_2 2021-11-20T06:02:08.331 INFO:journalctl@ceph.osd.4.smithi178.stdout:Nov 20 06:02:08 smithi178 conmon[48023]: debug 2021-11-20T06:02:08.073+0000 7f3e1938e080 -1 osd.4 0 log_to_monitors {default=true} 2021-11-20T06:02:08.332 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:08 smithi178 conmon[36600]: cluster 2021-11-20T06:02:06.194566+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:02:08.333 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:08 smithi178 conmon[36600]: 14162) 131 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:02:08.333 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:08 smithi178 conmon[36600]: audit 2021-11-20T06:02:07.193578+0000 mon.smithi139 (mon. 2021-11-20T06:02:08.334 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:08 smithi178 conmon[36600]: 0) 373 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:08.334 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:08 smithi178 conmon[36600]: audit 2021-11-20T06:02:08.334 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:08 smithi178 conmon[36600]: 2021-11-20T06:02:08. 2021-11-20T06:02:08.334 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:08 smithi178 conmon[36600]: 076742+0000 mon.smithi178 (mon.1) 10 : audit 2021-11-20T06:02:08.335 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:08 smithi178 conmon[36600]: [INF] from='osd.4 [v2:172.21.15.178:6808/2601954583,v1:172.21.15.178:6809/2601954583]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T06:02:08.335 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:08 smithi178 conmon[36600]: audit 2021-11-20T 2021-11-20T06:02:08.335 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:08 smithi178 conmon[36600]: 06:02:08.077044+0000 mon.smithi139 ( 2021-11-20T06:02:08.335 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:08 smithi178 conmon[36600]: mon.0) 374 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T06:02:08.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:08 smithi139 conmon[33501]: cluster 2021-11-20T06:02:06.194566+0000 mgr.smithi139.dplyxn (mgr.14162) 131 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:02:08.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:08 smithi139 conmon[33501]: audit 2021-11-20T06:02:07.193578+0000 mon.smithi139 (mon.0) 373 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:08.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:08 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:02:08.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:08 smithi139 conmon[33501]: 20T06:02:08.076742+0000 mon.smithi178 (mon.1) 10 : audit [INF] 2021-11-20T06:02:08.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:08 smithi139 conmon[33501]: from='osd.4 [v2:172.21.15.178:6808/2601954583,v1:172.21.15.178:6809/2601954583]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T06:02:08.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:08 smithi139 conmon[33501]: audit 2021-11-20T06: 2021-11-20T06:02:08.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:08 smithi139 conmon[33501]: 02:08.077044+0000 mon.smithi139 (mon.0 2021-11-20T06:02:08.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:08 smithi139 conmon[33501]: ) 374 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-20T06:02:08.868 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.877+0000 7f6f3f299700 1 -- 172.21.15.178:0/2225223540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f380f96a0 msgr2=0x7f6f380fbaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:08.869 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.877+0000 7f6f3f299700 1 --2- 172.21.15.178:0/2225223540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f380f96a0 0x7f6f380fbaf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f6f2c004660 tx=0x7f6f2c009b30).stop 2021-11-20T06:02:08.869 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.877+0000 7f6f3f299700 1 -- 172.21.15.178:0/2225223540 shutdown_connections 2021-11-20T06:02:08.870 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.877+0000 7f6f3f299700 1 --2- 172.21.15.178:0/2225223540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f380f96a0 0x7f6f380fbaf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:08.870 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.877+0000 7f6f3f299700 1 --2- 172.21.15.178:0/2225223540 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f38060270 0x7f6f380f91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:08.871 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.877+0000 7f6f3f299700 1 -- 172.21.15.178:0/2225223540 >> 172.21.15.178:0/2225223540 conn(0x7f6f380f0b90 msgr2=0x7f6f380f2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:08.872 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.877+0000 7f6f3f299700 1 -- 172.21.15.178:0/2225223540 shutdown_connections 2021-11-20T06:02:08.872 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.878+0000 7f6f3f299700 1 -- 172.21.15.178:0/2225223540 wait complete. 2021-11-20T06:02:08.873 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.878+0000 7f6f3f299700 1 Processor -- start 2021-11-20T06:02:08.874 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.878+0000 7f6f3f299700 1 -- start start 2021-11-20T06:02:08.874 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.878+0000 7f6f3f299700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f38060270 0x7f6f380fcff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:08.875 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.878+0000 7f6f3f299700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f380f96a0 0x7f6f380fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:08.875 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.878+0000 7f6f3f299700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380fdae0 con 0x7f6f380f96a0 2021-11-20T06:02:08.875 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.878+0000 7f6f3f299700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380fdc20 con 0x7f6f38060270 2021-11-20T06:02:08.876 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.878+0000 7f6f3e297700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f38060270 0x7f6f380fcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:08.876 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.878+0000 7f6f3e297700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f38060270 0x7f6f380fcff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:50222/0 (socket says 172.21.15.178:50222) 2021-11-20T06:02:08.877 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.878+0000 7f6f3e297700 1 -- 172.21.15.178:0/2870436137 learned_addr learned my addr 172.21.15.178:0/2870436137 (peer_addr_for_me v2:172.21.15.178:0/0) 2021-11-20T06:02:08.877 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.879+0000 7f6f3da96700 1 --2- 172.21.15.178:0/2870436137 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f380f96a0 0x7f6f380fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:08.877 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.879+0000 7f6f3e297700 1 -- 172.21.15.178:0/2870436137 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f380f96a0 msgr2=0x7f6f380fd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:08.878 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.879+0000 7f6f3e297700 1 --2- 172.21.15.178:0/2870436137 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f380f96a0 0x7f6f380fd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:08.878 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.879+0000 7f6f3e297700 1 -- 172.21.15.178:0/2870436137 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2c005040 con 0x7f6f38060270 2021-11-20T06:02:08.879 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.879+0000 7f6f3da96700 1 --2- 172.21.15.178:0/2870436137 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f380f96a0 0x7f6f380fd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:02:08.879 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.879+0000 7f6f3e297700 1 --2- 172.21.15.178:0/2870436137 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f38060270 0x7f6f380fcff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f6f2800c530 tx=0x7f6f28007e20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:08.880 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.879+0000 7f6f36ffd700 1 -- 172.21.15.178:0/2870436137 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f2800f650 con 0x7f6f38060270 2021-11-20T06:02:08.880 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.879+0000 7f6f36ffd700 1 -- 172.21.15.178:0/2870436137 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f2800fc60 con 0x7f6f38060270 2021-11-20T06:02:08.880 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.879+0000 7f6f3f299700 1 -- 172.21.15.178:0/2870436137 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f380fe590 con 0x7f6f38060270 2021-11-20T06:02:08.881 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.879+0000 7f6f36ffd700 1 -- 172.21.15.178:0/2870436137 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f28012c80 con 0x7f6f38060270 2021-11-20T06:02:08.882 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.879+0000 7f6f3f299700 1 -- 172.21.15.178:0/2870436137 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f380feb50 con 0x7f6f38060270 2021-11-20T06:02:08.882 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.880+0000 7f6f36ffd700 1 -- 172.21.15.178:0/2870436137 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6f28010070 con 0x7f6f38060270 2021-11-20T06:02:08.883 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.880+0000 7f6f3f299700 1 -- 172.21.15.178:0/2870436137 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f38059100 con 0x7f6f38060270 2021-11-20T06:02:08.884 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.880+0000 7f6f36ffd700 1 --2- 172.21.15.178:0/2870436137 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6f2005c300 0x7f6f2005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:08.885 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.881+0000 7f6f36ffd700 1 -- 172.21.15.178:0/2870436137 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3824+0+0 (secure 0 0 0) 0x7f6f28075510 con 0x7f6f38060270 2021-11-20T06:02:08.885 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.881+0000 7f6f3da96700 1 --2- 172.21.15.178:0/2870436137 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6f2005c300 0x7f6f2005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:08.885 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.881+0000 7f6f3da96700 1 --2- 172.21.15.178:0/2870436137 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6f2005c300 0x7f6f2005e7c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f6f38060040 tx=0x7f6f2c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:08.886 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:08.884+0000 7f6f36ffd700 1 -- 172.21.15.178:0/2870436137 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f280452b0 con 0x7f6f38060270 2021-11-20T06:02:09.062 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:09.076+0000 7f6f3f299700 1 -- 172.21.15.178:0/2870436137 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f380feef0 con 0x7f6f2005c300 2021-11-20T06:02:09.442 INFO:journalctl@ceph.osd.4.smithi178.stdout:Nov 20 06:02:09 smithi178 conmon[48023]: debug 2021-11-20T06:02:09.199+0000 7f3e0f8ff700 -1 osd.4 0 waiting for initial osdmap 2021-11-20T06:02:09.442 INFO:journalctl@ceph.osd.4.smithi178.stdout:Nov 20 06:02:09 smithi178 conmon[48023]: debug 2021-11-20T06:02:09.207+0000 7f3e0b87a700 -1 osd.4 30 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T06:02:09.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:09 smithi178 conmon[36600]: cluster 2021-11-20T06:02:08.194833+0000 mgr.smithi139.dplyxn (mgr.14162) 132 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:02:09.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:09 smithi178 conmon[36600]: audit 2021-11-20T06:02:08.194901+0000 mon.smithi139 (mon.0) 375 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-11-20T06:02:09.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:09 smithi178 conmon[36600]: cluster 2021-11-20T06:02:08.194971+0000 mon.smithi139 (mon.0) 376 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-11-20T06:02:09.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:09 smithi178 conmon[36600]: audit 2021-11-20T06:02:08.195068+0000 mon.smithi139 (mon.0) 377 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T06:02:09.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:09 smithi178 conmon[36600]: audit 2021-11-20T06:02:08.195609+0000 mon.smithi178 (mon.1) 11 : audit [INF] from='osd.4 [v2:172.21.15.178:6808/2601954583,v1:172.21.15.178:6809/2601954583]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2021-11-20T06:02:09.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:09 smithi178 conmon[36600]: audit 2021-11-20T06:02:08.195852+0000 mon.smithi139 (mon.0) 378 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2021-11-20T06:02:09.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:09 smithi178 conmon[36600]: audit 2021-11-20T06:02:08.742291+0000 mon.smithi139 (mon.0) 379 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:09.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:09 smithi178 conmon[36600]: audit 2021-11-20T06:02:09.078177+0000 mgr.smithi139.dplyxn (mgr.14162) 133 : audit [DBG] from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:02:09.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:09 smithi178 conmon[36600]: audit 2021-11-20T06:02:09.078918+0000 mon.smithi139 (mon.0) 380 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T06:02:09.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:09 smithi178 conmon[36600]: cephadm 2021-11-20T06:02:09.079967+0000 mgr.smithi139.dplyxn (mgr.14162) 134 : cephadm [INF] Applying service osd.None on host smithi178... 2021-11-20T06:02:09.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:09 smithi178 conmon[36600]: audit 2021-11-20T06:02:09.080172+0000 mon.smithi139 (mon.0) 381 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T06:02:09.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:09 smithi178 conmon[36600]: audit 2021-11-20T06:02:09.080700+0000 mon.smithi139 (mon.0) 382 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:02:09.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: cluster 2021-11-20T06:02:08.194833+0000 mgr.smithi139.dplyxn (mgr.14162) 132 : 2021-11-20T06:02:09.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:02:09.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: audit 2021-11-20T06:02:08.194901+0000 mon.smithi139 (mon.0) 375 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-11-20T06:02:09.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: cluster 2021-11-20T06:02:08.194971+0000 mon.smithi139 (mon. 2021-11-20T06:02:09.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: 0) 376 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-11-20T06:02:09.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: audit 2021-11-20T06:02:08.195068+0000 mon.smithi139 (mon.0) 377 2021-11-20T06:02:09.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T06:02:09.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:02:09.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: 20T06:02:08.195609+0000 mon.smithi178 (mon.1) 11 : audit [INF] from='osd.4 [v2:172.21.15.178:6808/2601954583,v1:172.21.15.178:6809/2601954583]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2021-11-20T06:02:09.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: audit 2021-11-20T06:02:08.195852+0000 2021-11-20T06:02:09.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: mon.smithi139 (mon.0) 378 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2021-11-20T06:02:09.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: audit 2021-11-20T06:02:08.742291+0000 mon.smithi139 (mon. 2021-11-20T06:02:09.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: 0) 379 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:09.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: audit 2021-11-20T06:02:09.078177+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:02:09.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: mgr.14162) 133 : audit [DBG] from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:02:09.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: audit 2021-11-20T 2021-11-20T06:02:09.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: 06:02:09.078918+0000 mon.smithi139 (mon.0) 380 : audit 2021-11-20T06:02:09.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-20T06:02:09.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: cephadm 2021-11-20T06:02:09.079967+0000 mgr.smithi139.dplyxn 2021-11-20T06:02:09.534 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: (mgr.14162) 134 : cephadm [INF] Applying service osd.None on host smithi178... 2021-11-20T06:02:09.534 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: audit 2021-11-20T06:02:09.080172 2021-11-20T06:02:09.534 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 381 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-20T06:02:09.535 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: audit 2021-11-20T06: 2021-11-20T06:02:09.535 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:09 smithi139 conmon[33501]: 02:09.080700+0000 mon.smithi139 (mon.0) 382 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:02:10.389 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:10 smithi178 conmon[36600]: audit 2021-11-20T06:02:09.196507+0000 mon.smithi139 (mon.0) 383 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2021-11-20T06:02:10.389 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:10 smithi178 conmon[36600]: cluster 2021-11-20T06:02:09.196576+0000 mon.smithi139 (mon.0) 384 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2021-11-20T06:02:10.389 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:10 smithi178 conmon[36600]: audit 2021-11-20T06:02:09.196930+0000 mon.smithi139 (mon.0) 385 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T06:02:10.390 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:10 smithi178 conmon[36600]: audit 2021-11-20T06:02:09.200249+0000 mon.smithi139 (mon.0) 386 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T06:02:10.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:10 smithi139 conmon[33501]: audit 2021-11-20T06:02:09.196507+0000 mon.smithi139 (mon.0) 383 : 2021-11-20T06:02:10.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:10 smithi139 conmon[33501]: audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2021-11-20T06:02:10.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:10 smithi139 conmon[33501]: cluster 2021-11-20T06:02:09.196576+0000 mon.smithi139 (mon.0) 384 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2021-11-20T06:02:10.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:10 smithi139 conmon[33501]: audit 2021-11-20T06: 2021-11-20T06:02:10.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:10 smithi139 conmon[33501]: 02:09.196930+0000 mon.smithi139 (mon.0) 385 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T06:02:10.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:10 smithi139 conmon[33501]: audit 2021-11-20T06:02:09.200249+0000 mon.smithi139 (mon.0 2021-11-20T06:02:10.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:10 smithi139 conmon[33501]: ) 386 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T06:02:11.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:11 smithi178 conmon[36600]: cluster 2021-11-20T06:02:09.060385+0000 osd.4 ( 2021-11-20T06:02:11.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:11 smithi178 conmon[36600]: osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T06:02:11.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:11 smithi178 conmon[36600]: cluster 2021-11-20T06:02:09.060489+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T06:02:11.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:11 smithi178 conmon[36600]: cluster 2021-11-20T06:02:10.195161+0000 mgr.smithi139.dplyxn (mgr.14162) 135 : cluster [DBG] pgmap v101: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:02:11.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:11 smithi178 conmon[36600]: audit 2021-11-20T06:02:10.201817+0000 mon.smithi139 (mon.0) 387 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T06:02:11.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:11 smithi178 conmon[36600]: cluster 2021-11-20T06:02:10.209860+0000 mon.smithi139 (mon.0) 388 : cluster [INF] osd.4 [v2:172.21.15.178:6808/2601954583,v1:172.21.15.178:6809/2601954583] boot 2021-11-20T06:02:11.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:11 smithi178 conmon[36600]: cluster 2021-11-20T06:02:10.209935+0000 mon.smithi139 (mon.0) 389 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-11-20T06:02:11.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:11 smithi178 conmon[36600]: audit 2021-11-20T06:02:10.210273+0000 mon.smithi139 (mon.0) 390 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T06:02:11.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:11 smithi178 conmon[36600]: cephadm 2021-11-20T06:02:10.755934+0000 mgr.smithi139.dplyxn (mgr.14162) 136 : cephadm [INF] Detected new or changed devices on smithi178 2021-11-20T06:02:11.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:11 smithi178 conmon[36600]: audit 2021-11-20T06:02:10.761198+0000 mon.smithi139 (mon.0) 391 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:11.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:11 smithi178 conmon[36600]: audit 2021-11-20T06:02:10.764736+0000 mon.smithi139 (mon.0) 392 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:11.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:11 smithi178 conmon[36600]: audit 2021-11-20T06:02:10.767608+0000 mon.smithi139 (mon.0) 393 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:02:11.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: cluster 2021-11-20T06:02:09.060385+0000 osd.4 (osd.4) 2021-11-20T06:02:11.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T06:02:11.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: cluster 2021-11-20T06:02:09.060489+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T06:02:11.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: cluster 2021-11 2021-11-20T06:02:11.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: -20T06:02:10.195161+0000 mgr.smithi139.dplyxn (mgr.14162) 135 : cluster [DBG] pgmap v101: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-20T06:02:11.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: audit 2021-11 2021-11-20T06:02:11.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: -20T06:02:10.201817+0000 mon.smithi139 ( 2021-11-20T06:02:11.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: mon.0) 387 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T06:02:11.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: cluster 2021-11- 2021-11-20T06:02:11.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: 20T06:02:10.209860+0000 mon.smithi139 (mon.0 2021-11-20T06:02:11.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: ) 388 : cluster [INF] osd.4 [v2:172.21.15.178:6808/2601954583,v1:172.21.15.178:6809/2601954583] boot 2021-11-20T06:02:11.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: cluster 2021-11-20T06 2021-11-20T06:02:11.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: :02:10.209935+0000 mon.smithi139 (mon.0) 389 2021-11-20T06:02:11.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-11-20T06:02:11.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: audit 2021-11 2021-11-20T06:02:11.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: -20T06:02:10.210273+0000 mon.smithi139 (mon.0) 390 2021-11-20T06:02:11.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-20T06:02:11.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: cephadm 2021-11 2021-11-20T06:02:11.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: -20T06:02:10.755934+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:02:11.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: 14162) 136 : cephadm [INF] Detected new or changed devices on smithi178 2021-11-20T06:02:11.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: audit 2021-11-20T06:02:11.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: 2021-11-20T06:02: 2021-11-20T06:02:11.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: 10.761198+0000 mon.smithi139 (mon. 2021-11-20T06:02:11.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: 0) 391 : audit [INF] 2021-11-20T06:02:11.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:11.534 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: audit 2021-11-20T06:02:11.534 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: 2021-11-20T 2021-11-20T06:02:11.534 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: 06:02:10.764736+0000 2021-11-20T06:02:11.534 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: mon.smithi139 (mon.0) 392 : audit [INF] 2021-11-20T06:02:11.535 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:11.535 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: audit 2021-11-20 2021-11-20T06:02:11.535 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: T06:02:10.767608+0000 mon.smithi139 2021-11-20T06:02:11.535 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: (mon.0) 393 : audit 2021-11-20T06:02:11.535 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:11 smithi139 conmon[33501]: [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:02:12.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: audit 2021-11-20T06:02:11.371609+0000 mon.smithi178 2021-11-20T06:02:12.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: (mon.1) 12 : audit [INF] from='client.? 172.21.15.178:0/4072655990' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7d0eee4-3cb1-43dc-9c3f-f95593cf47cb"}]: dispatch 2021-11-20T06:02:12.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: audit 2021-11-20T06 2021-11-20T06:02:12.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: :02:11.371950+0000 mon.smithi139 (mon.0) 394 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7d0eee4-3cb1-43dc-9c3f-f95593cf47cb"}]: dispatch 2021-11-20T06:02:12.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: audit 2021-11-20T06:02:12.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: 2021-11-20T06:02:11 2021-11-20T06:02:12.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: .375820+0000 mon.smithi139 (mon.0) 395 : audit 2021-11-20T06:02:12.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7d0eee4-3cb1-43dc-9c3f-f95593cf47cb"}]': finished 2021-11-20T06:02:12.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: cluster 2021-11-20 2021-11-20T06:02:12.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: T06:02:11.375867+0000 mon.smithi139 ( 2021-11-20T06:02:12.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: mon.0) 396 : cluster [DBG] osdmap e32: 6 total, 5 up, 6 in 2021-11-20T06:02:12.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: audit 2021-11-20T06:02:12.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: 2021-11-20T06:02: 2021-11-20T06:02:12.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: 11.375974+0000 mon.smithi139 (mon.0) 397 : 2021-11-20T06:02:12.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T06:02:12.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: audit 2021-11-20 2021-11-20T06:02:12.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: T06:02:11.788910+0000 mon.smithi178 ( 2021-11-20T06:02:12.447 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:12 smithi178 conmon[36600]: mon.1) 13 : audit [DBG] from='client.? 172.21.15.178:0/1452904589' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T06:02:12.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: audit 2021-11-20T06:02:11.371609+0000 2021-11-20T06:02:12.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: mon.smithi178 (mon.1) 12 : audit [INF] from='client.? 172.21.15.178:0/4072655990' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7d0eee4-3cb1-43dc-9c3f-f95593cf47cb"}]: dispatch 2021-11-20T06:02:12.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: audit 2021-11-20 2021-11-20T06:02:12.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: T06:02:11.371950+0000 mon.smithi139 (mon 2021-11-20T06:02:12.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: .0) 394 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7d0eee4-3cb1-43dc-9c3f-f95593cf47cb"}]: dispatch 2021-11-20T06:02:12.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: 2021-11-20T06:02:12.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: audit 2021 2021-11-20T06:02:12.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: -11-20T06:02:11 2021-11-20T06:02:12.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: .375820+0000 mon.smithi139 (mon.0) 2021-11-20T06:02:12.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: 395 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7d0eee4-3cb1-43dc-9c3f-f95593cf47cb"}]': finished 2021-11-20T06:02:12.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: cluster 2021- 2021-11-20T06:02:12.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: 11-20T06:02:11.375867 2021-11-20T06:02:12.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: +0000 mon.smithi139 ( 2021-11-20T06:02:12.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: mon.0 2021-11-20T06:02:12.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: ) 396 : cluster [DBG] osdmap e32: 6 total, 5 up, 6 in 2021-11-20T06:02:12.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: audit 2021- 2021-11-20T06:02:12.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: 11-20T06:02:11.375974 2021-11-20T06:02:12.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 397 : 2021-11-20T06:02:12.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T06:02:12.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: audit 2021-11 2021-11-20T06:02:12.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: -20T06:02:11.788910+0000 mon.smithi178 ( 2021-11-20T06:02:12.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:12 smithi139 conmon[33501]: mon.1) 13 : audit [DBG] from='client.? 172.21.15.178:0/1452904589' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-20T06:02:14.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:14 smithi178 conmon[36600]: cluster 2021-11-20T06:02:12.195466+0000 mgr.smithi139.dplyxn (mgr.14162) 137 : cluster [DBG] pgmap v104: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:14.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:14 smithi139 conmon[33501]: cluster 2021-11-20T06:02:12.195466+0000 mgr.smithi139.dplyxn (mgr.14162) 137 2021-11-20T06:02:14.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:14 smithi139 conmon[33501]: : cluster [DBG] pgmap v104: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:16.192 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:16 smithi178 conmon[36600]: cluster 2021-11-20T06:02:14.195918+0000 mgr.smithi139.dplyxn (mgr.14162) 138 : cluster [DBG] pgmap v105: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:16.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:16 smithi178 conmon[36600]: audit 2021-11-20T06:02:15.470194+0000 mon.smithi139 (mon.0) 398 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:16.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:16 smithi178 conmon[36600]: audit 2021-11-20T06:02:15.473146+0000 mon.smithi139 (mon.0) 399 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:16.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:16 smithi139 conmon[33501]: cluster 2021-11-20T06:02:14.195918+0000 mgr.smithi139.dplyxn (mgr.14162) 138 : cluster [DBG] 2021-11-20T06:02:16.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:16 smithi139 conmon[33501]: pgmap v105: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:16.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:16 smithi139 conmon[33501]: audit 2021-11-20T06:02:15.470194+0000 mon.smithi139 (mon.0) 398 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:16.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:16 smithi139 conmon[33501]: audit 2021-11-20T06:02:15.473146+0000 mon.smithi139 (mon. 2021-11-20T06:02:16.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:16 smithi139 conmon[33501]: 0) 399 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:18.407 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:18 smithi178 conmon[36600]: cluster 2021-11-20T06:02:16.196369+0000 mgr.smithi139.dplyxn (mgr.14162) 139 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:18.407 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:18 smithi178 conmon[36600]: audit 2021-11-20T06:02:17.744145+0000 mon.smithi139 (mon.0) 400 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-11-20T06:02:18.408 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:18 smithi178 conmon[36600]: audit 2021-11-20T06:02:17.745035+0000 mon.smithi139 (mon.0) 401 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:02:18.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:18 smithi139 conmon[33501]: cluster 2021-11-20T06:02:16.196369+0000 mgr.smithi139.dplyxn (mgr.14162) 139 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:18.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:18 smithi139 conmon[33501]: 2021-11-20T06:02:18.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:18 smithi139 conmon[33501]: audit 2021-11-20T06:02:17.744145+0000 mon.smithi139 (mon.0) 400 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-11-20T06:02:18.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:18 smithi139 conmon[33501]: audit 2021-11-20T06:02:17.745035+0000 mon.smithi139 (mon.0) 401 : audit [DBG] 2021-11-20T06:02:18.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:18 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:02:19.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:19 smithi178 conmon[36600]: cephadm 2021-11-20T06:02:17.745777+0000 mgr.smithi139.dplyxn (mgr.14162) 140 : cephadm [INF] Deploying daemon osd.5 on smithi178 2021-11-20T06:02:19.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:19 smithi139 conmon[33501]: cephadm 2021-11-20T06:02:17.745777+0000 mgr.smithi139.dplyxn (mgr.14162) 140 : cephadm [INF] Deploying daemon osd.5 on smithi178 2021-11-20T06:02:20.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:20 smithi178 conmon[36600]: cluster 2021-11-20T06:02:18.196747+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:02:20.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:20 smithi178 conmon[36600]: 141 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:20.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:20 smithi139 conmon[33501]: cluster 2021-11-20T06:02:18.196747+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:02:20.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:20 smithi139 conmon[33501]: 141 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:20.845 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 5 on host 'smithi178' 2021-11-20T06:02:20.846 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:20.857+0000 7f6f36ffd700 1 -- 172.21.15.178:0/2870436137 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6f380feef0 con 0x7f6f2005c300 2021-11-20T06:02:20.846 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:20.859+0000 7f6f3f299700 1 -- 172.21.15.178:0/2870436137 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6f2005c300 msgr2=0x7f6f2005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:20.846 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:20.859+0000 7f6f3f299700 1 --2- 172.21.15.178:0/2870436137 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6f2005c300 0x7f6f2005e7c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f6f38060040 tx=0x7f6f2c009b30).stop 2021-11-20T06:02:20.847 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:20.859+0000 7f6f3f299700 1 -- 172.21.15.178:0/2870436137 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f38060270 msgr2=0x7f6f380fcff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:20.847 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:20.859+0000 7f6f3f299700 1 --2- 172.21.15.178:0/2870436137 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f38060270 0x7f6f380fcff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f6f2800c530 tx=0x7f6f28007e20).stop 2021-11-20T06:02:20.847 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:20.859+0000 7f6f3f299700 1 -- 172.21.15.178:0/2870436137 shutdown_connections 2021-11-20T06:02:20.849 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:20.859+0000 7f6f3f299700 1 --2- 172.21.15.178:0/2870436137 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6f2005c300 0x7f6f2005e7c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:20.849 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:20.859+0000 7f6f3f299700 1 --2- 172.21.15.178:0/2870436137 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f38060270 0x7f6f380fcff0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:20.849 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:20.859+0000 7f6f3f299700 1 --2- 172.21.15.178:0/2870436137 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6f380f96a0 0x7f6f380fd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:20.849 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:20.859+0000 7f6f3f299700 1 -- 172.21.15.178:0/2870436137 >> 172.21.15.178:0/2870436137 conn(0x7f6f380f0b90 msgr2=0x7f6f380f2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:20.850 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:20.859+0000 7f6f3f299700 1 -- 172.21.15.178:0/2870436137 shutdown_connections 2021-11-20T06:02:20.850 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:20.859+0000 7f6f3f299700 1 -- 172.21.15.178:0/2870436137 wait complete. 2021-11-20T06:02:21.222 DEBUG:teuthology.orchestra.run.smithi178:osd.5> sudo journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.5.service 2021-11-20T06:02:21.225 INFO:tasks.cephadm:Setting up client nodes... 2021-11-20T06:02:21.226 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-11-20T06:02:21.253 INFO:journalctl@ceph.osd.5.smithi178.stdout:-- Logs begin at Sat 2021-11-20 05:48:12 UTC. -- 2021-11-20T06:02:21.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:21 smithi178 conmon[36600]: cluster 2021-11-20T06:02:20.197251+0000 mgr.smithi139.dplyxn (mgr.14162) 142 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:21.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:21 smithi178 conmon[36600]: audit 2021-11-20T06:02:20.839373+0000 mon.smithi139 (mon.0) 402 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:02:21.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:21 smithi178 conmon[36600]: audit 2021-11-20T06:02:20.844193+0000 mon.smithi139 (mon.0) 403 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:21.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:21 smithi178 conmon[36600]: audit 2021-11-20T06:02:20.857203+0000 mon.smithi139 (mon.0) 404 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:21.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:21 smithi139 conmon[33501]: cluster 2021-11-20T06:02:20.197251+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:02:21.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:21 smithi139 conmon[33501]: ) 142 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:21.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:21 smithi139 conmon[33501]: audit 2021-11-20T06:02:20.839373+0000 mon.smithi139 (mon.0) 402 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:02:21.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:21 smithi139 conmon[33501]: audit 2021-11-20T06:02:20.844193+0000 mon.smithi139 (mon.0 2021-11-20T06:02:21.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:21 smithi139 conmon[33501]: ) 403 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:21.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:21 smithi139 conmon[33501]: audit 2021-11-20T06 2021-11-20T06:02:21.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:21 smithi139 conmon[33501]: :02:20.857203+0000 mon.smithi139 (mon.0) 404 2021-11-20T06:02:21.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:21 smithi139 conmon[33501]: : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:22.917 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.928+0000 7f90472b5700 1 -- 172.21.15.139:0/2204561128 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90400f77d0 msgr2=0x7f90400f7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:23.130 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.928+0000 7f90472b5700 1 --2- 172.21.15.139:0/2204561128 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90400f77d0 0x7f90400f7c30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f9030004660 tx=0x7f9030009b30).stop 2021-11-20T06:02:23.130 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.928+0000 7f90472b5700 1 -- 172.21.15.139:0/2204561128 shutdown_connections 2021-11-20T06:02:23.130 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.928+0000 7f90472b5700 1 --2- 172.21.15.139:0/2204561128 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90400f77d0 0x7f90400f7c30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:23.131 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.928+0000 7f90472b5700 1 --2- 172.21.15.139:0/2204561128 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90400f6530 0x7f90400f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:02:23.131 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.928+0000 7f90472b5700 1 -- 172.21.15.139:0/2204561128 >> 172.21.15.139:0/2204561128 conn(0x7f90400f1cd0 msgr2=0x7f90400f40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:23.131 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.928+0000 7f90472b5700 1 -- 172.21.15.139:0/2204561128 shutdown_connections 2021-11-20T06:02:23.131 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.928+0000 7f90472b5700 1 -- 172.21.15.139:0/2204561128 wait complete. 2021-11-20T06:02:23.131 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.929+0000 7f90472b5700 1 Processor -- start 2021-11-20T06:02:23.132 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.929+0000 7f90472b5700 1 -- start start 2021-11-20T06:02:23.132 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.929+0000 7f90472b5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90400f6530 0x7f904006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:23.132 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.929+0000 7f90472b5700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90400f77d0 0x7f9040069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:23.132 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.929+0000 7f90472b5700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904006a410 con 0x7f90400f6530 2021-11-20T06:02:23.133 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.929+0000 7f90472b5700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904006a550 con 0x7f90400f77d0 2021-11-20T06:02:23.133 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.929+0000 7f90462b3700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90400f6530 0x7f904006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:23.133 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.929+0000 7f90462b3700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90400f6530 0x7f904006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57090/0 (socket says 172.21.15.139:57090) 2021-11-20T06:02:23.133 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.929+0000 7f90462b3700 1 -- 172.21.15.139:0/2334327729 learned_addr learned my addr 172.21.15.139:0/2334327729 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:23.133 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.929+0000 7f9045ab2700 1 --2- 172.21.15.139:0/2334327729 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90400f77d0 0x7f9040069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:23.134 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.930+0000 7f90462b3700 1 -- 172.21.15.139:0/2334327729 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90400f77d0 msgr2=0x7f9040069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:23.134 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.930+0000 7f90462b3700 1 --2- 172.21.15.139:0/2334327729 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90400f77d0 0x7f9040069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:23.134 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.930+0000 7f90462b3700 1 -- 172.21.15.139:0/2334327729 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9030005040 con 0x7f90400f6530 2021-11-20T06:02:23.134 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.930+0000 7f90462b3700 1 --2- 172.21.15.139:0/2334327729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90400f6530 0x7f904006b9a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f903c0093f0 tx=0x7f903c006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:23.135 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.930+0000 7f9036ffd700 1 -- 172.21.15.139:0/2334327729 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f903c013640 con 0x7f90400f6530 2021-11-20T06:02:23.135 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.930+0000 7f90472b5700 1 -- 172.21.15.139:0/2334327729 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904006a830 con 0x7f90400f6530 2021-11-20T06:02:23.135 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.930+0000 7f9036ffd700 1 -- 172.21.15.139:0/2334327729 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f903c013c50 con 0x7f90400f6530 2021-11-20T06:02:23.135 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.930+0000 7f9036ffd700 1 -- 172.21.15.139:0/2334327729 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f903c012c80 con 0x7f90400f6530 2021-11-20T06:02:23.135 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.930+0000 7f90472b5700 1 -- 172.21.15.139:0/2334327729 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f904010ed80 con 0x7f90400f6530 2021-11-20T06:02:23.137 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.931+0000 7f90472b5700 1 -- 172.21.15.139:0/2334327729 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9040045ad0 con 0x7f90400f6530 2021-11-20T06:02:23.137 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.932+0000 7f9036ffd700 1 -- 172.21.15.139:0/2334327729 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f903c01a060 con 0x7f90400f6530 2021-11-20T06:02:23.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.932+0000 7f9036ffd700 1 --2- 172.21.15.139:0/2334327729 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f902c05c300 0x7f902c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:23.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.932+0000 7f9036ffd700 1 -- 172.21.15.139:0/2334327729 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4235+0+0 (secure 0 0 0) 0x7f903c075e20 con 0x7f90400f6530 2021-11-20T06:02:23.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.932+0000 7f9045ab2700 1 --2- 172.21.15.139:0/2334327729 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f902c05c300 0x7f902c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:23.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.933+0000 7f9045ab2700 1 --2- 172.21.15.139:0/2334327729 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f902c05c300 0x7f902c05e7c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f9030004630 tx=0x7f9030009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:23.139 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:22.936+0000 7f9036ffd700 1 -- 172.21.15.139:0/2334327729 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f903c049050 con 0x7f90400f6530 2021-11-20T06:02:23.141 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:23.154+0000 7f90472b5700 1 -- 172.21.15.139:0/2334327729 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f9040027a00 con 0x7f90400f6530 2021-11-20T06:02:23.145 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:23.159+0000 7f9036ffd700 1 -- 172.21.15.139:0/2334327729 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f903c04c670 con 0x7f90400f6530 2021-11-20T06:02:23.145 INFO:teuthology.orchestra.run.smithi139.stdout:[client.0] 2021-11-20T06:02:23.146 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQBvj5hh93hfCRAAaMsf9nQqYh90uDoF5Oi21Q== 2021-11-20T06:02:23.147 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:23.161+0000 7f90472b5700 1 -- 172.21.15.139:0/2334327729 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f902c05c300 msgr2=0x7f902c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:23.148 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:23.161+0000 7f90472b5700 1 --2- 172.21.15.139:0/2334327729 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f902c05c300 0x7f902c05e7c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f9030004630 tx=0x7f9030009b30).stop 2021-11-20T06:02:23.148 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:23.161+0000 7f90472b5700 1 -- 172.21.15.139:0/2334327729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90400f6530 msgr2=0x7f904006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:23.148 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:23.161+0000 7f90472b5700 1 --2- 172.21.15.139:0/2334327729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90400f6530 0x7f904006b9a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f903c0093f0 tx=0x7f903c006ec0).stop 2021-11-20T06:02:23.148 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:23.161+0000 7f90472b5700 1 -- 172.21.15.139:0/2334327729 shutdown_connections 2021-11-20T06:02:23.148 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:23.161+0000 7f90472b5700 1 --2- 172.21.15.139:0/2334327729 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f902c05c300 0x7f902c05e7c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:23.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:23.161+0000 7f90472b5700 1 --2- 172.21.15.139:0/2334327729 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90400f6530 0x7f904006b9a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:23.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:23.161+0000 7f90472b5700 1 --2- 172.21.15.139:0/2334327729 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90400f77d0 0x7f9040069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:23.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:23.161+0000 7f90472b5700 1 -- 172.21.15.139:0/2334327729 >> 172.21.15.139:0/2334327729 conn(0x7f90400f1cd0 msgr2=0x7f90400fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:23.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:23.161+0000 7f90472b5700 1 -- 172.21.15.139:0/2334327729 shutdown_connections 2021-11-20T06:02:23.150 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:23.161+0000 7f90472b5700 1 -- 172.21.15.139:0/2334327729 wait complete. 2021-11-20T06:02:23.170 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:23 smithi139 conmon[33501]: audit 2021-11-20T06:02:23.157064+0000 mon.smithi139 (mon.0) 405 : audit [INF] from='client.? 172.21.15.139:0/2334327729' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T06:02:23.170 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:23 smithi139 conmon[33501]: audit 2021-11-20T06:02:23.160514+0000 mon.smithi139 (mon.0) 406 : audit [INF] from='client.? 172.21.15.139:0/2334327729' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T06:02:23.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:23 smithi178 conmon[36600]: audit 2021-11-20T06:02:23.157064+0000 mon.smithi139 (mon.0) 2021-11-20T06:02:23.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:23 smithi178 conmon[36600]: 405 : audit [INF] from='client.? 172.21.15.139:0/2334327729' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T06:02:23.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:23 smithi178 conmon[36600]: audit 2021-11-20T06:02:23.160514+0000 mon.smithi139 (mon.0) 406 : audit [INF] from='client.? 172.21.15.139:0/2334327729' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T06:02:23.463 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2021-11-20T06:02:23.464 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2021-11-20T06:02:23.464 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-11-20T06:02:23.509 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-11-20T06:02:24.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:24 smithi178 conmon[36600]: cluster 2021-11-20T06:02:22.197703+0000 mgr.smithi139.dplyxn (mgr.14162) 143 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:24.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:24 smithi178 conmon[36600]: audit 2021-11-20T06:02:23.477752+0000 mon.smithi139 (mon.0) 407 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:24.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:24 smithi139 conmon[33501]: cluster 2021-11-20T06:02:22.197703+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:02:24.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:24 smithi139 conmon[33501]: 143 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:24.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:24 smithi139 conmon[33501]: audit 2021-11-20T06:02:23.477752+0000 mon.smithi139 (mon.0) 407 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:25.078 INFO:journalctl@ceph.osd.5.smithi178.stdout:Nov 20 06:02:24 smithi178 conmon[52933]: debug 2021-11-20T06:02:24.766+0000 7fbf1bb97080 -1 osd.5 0 log_to_monitors {default=true} 2021-11-20T06:02:25.380 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:25 smithi178 conmon[36600]: audit 2021-11-20T06:02:24.770206+0000 mon.smithi178 (mon.1) 14 : audit [INF] from='osd.5 [v2:172.21.15.178:6816/3932451665,v1:172.21.15.178:6817/3932451665]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T06:02:25.380 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:25 smithi178 conmon[36600]: audit 2021-11-20T06:02:24.770733+0000 mon.smithi139 (mon.0) 408 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T06:02:25.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:25 smithi139 conmon[33501]: audit 2021-11-20T06:02:24.770206+0000 mon.smithi178 (mon.1) 14 : audit [INF] from='osd.5 [v2:172.21.15.178:6816/3932451665,v1:172.21.15.178:6817/3932451665]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T06:02:25.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:25 smithi139 conmon[33501]: audit 2021-11-20T06:02:24.770733+0000 mon.smithi139 (mon.0) 408 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-20T06:02:25.644 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.657+0000 7f2d24526700 1 -- 172.21.15.178:0/2652489041 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d1c0f5640 msgr2=0x7f2d1c0f5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:25.645 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.657+0000 7f2d24526700 1 --2- 172.21.15.178:0/2652489041 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d1c0f5640 0x7f2d1c0f5a60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f2d18004660 tx=0x7f2d18009b30).stop 2021-11-20T06:02:25.645 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.657+0000 7f2d24526700 1 -- 172.21.15.178:0/2652489041 shutdown_connections 2021-11-20T06:02:25.645 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.657+0000 7f2d24526700 1 --2- 172.21.15.178:0/2652489041 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d1c0f68e0 0x7f2d1c0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:25.646 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.657+0000 7f2d24526700 1 --2- 172.21.15.178:0/2652489041 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d1c0f5640 0x7f2d1c0f5a60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:25.646 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.657+0000 7f2d24526700 1 -- 172.21.15.178:0/2652489041 >> 172.21.15.178:0/2652489041 conn(0x7f2d1c0f0da0 msgr2=0x7f2d1c0f31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:25.646 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.657+0000 7f2d24526700 1 -- 172.21.15.178:0/2652489041 shutdown_connections 2021-11-20T06:02:25.646 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.657+0000 7f2d24526700 1 -- 172.21.15.178:0/2652489041 wait complete. 2021-11-20T06:02:25.647 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.657+0000 7f2d24526700 1 Processor -- start 2021-11-20T06:02:25.647 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d24526700 1 -- start start 2021-11-20T06:02:25.647 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d24526700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d1c0f5640 0x7f2d1c10c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:25.647 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d24526700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d1c0f68e0 0x7f2d1c10e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:25.647 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d24526700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c060490 con 0x7f2d1c0f68e0 2021-11-20T06:02:25.648 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d24526700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c0605d0 con 0x7f2d1c0f5640 2021-11-20T06:02:25.648 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d222c2700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d1c0f5640 0x7f2d1c10c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:25.648 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d222c2700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d1c0f5640 0x7f2d1c10c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:50298/0 (socket says 172.21.15.178:50298) 2021-11-20T06:02:25.649 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d222c2700 1 -- 172.21.15.178:0/4149773904 learned_addr learned my addr 172.21.15.178:0/4149773904 (peer_addr_for_me v2:172.21.15.178:0/0) 2021-11-20T06:02:25.649 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d21ac1700 1 --2- 172.21.15.178:0/4149773904 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d1c0f68e0 0x7f2d1c10e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:25.650 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d222c2700 1 -- 172.21.15.178:0/4149773904 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d1c0f68e0 msgr2=0x7f2d1c10e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:25.650 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d222c2700 1 --2- 172.21.15.178:0/4149773904 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d1c0f68e0 0x7f2d1c10e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:25.650 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d222c2700 1 -- 172.21.15.178:0/4149773904 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d18005040 con 0x7f2d1c0f5640 2021-11-20T06:02:25.651 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d21ac1700 1 --2- 172.21.15.178:0/4149773904 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d1c0f68e0 0x7f2d1c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:02:25.651 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d222c2700 1 --2- 172.21.15.178:0/4149773904 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d1c0f5640 0x7f2d1c10c0e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f2d18009f50 tx=0x7f2d18009b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:25.651 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d12ffd700 1 -- 172.21.15.178:0/4149773904 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d18018070 con 0x7f2d1c0f5640 2021-11-20T06:02:25.651 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d24526700 1 -- 172.21.15.178:0/4149773904 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1c10ead0 con 0x7f2d1c0f5640 2021-11-20T06:02:25.652 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d12ffd700 1 -- 172.21.15.178:0/4149773904 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d18013410 con 0x7f2d1c0f5640 2021-11-20T06:02:25.652 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d12ffd700 1 -- 172.21.15.178:0/4149773904 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d1801c620 con 0x7f2d1c0f5640 2021-11-20T06:02:25.652 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.658+0000 7f2d24526700 1 -- 172.21.15.178:0/4149773904 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1c10eea0 con 0x7f2d1c0f5640 2021-11-20T06:02:25.653 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.659+0000 7f2d24526700 1 -- 172.21.15.178:0/4149773904 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d1c0f4a30 con 0x7f2d1c0f5640 2021-11-20T06:02:25.654 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.660+0000 7f2d12ffd700 1 -- 172.21.15.178:0/4149773904 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2d1801d420 con 0x7f2d1c0f5640 2021-11-20T06:02:25.654 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.660+0000 7f2d12ffd700 1 --2- 172.21.15.178:0/4149773904 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f2d0805c2b0 0x7f2d0805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:25.654 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.660+0000 7f2d12ffd700 1 -- 172.21.15.178:0/4149773904 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4256+0+0 (secure 0 0 0) 0x7f2d180778f0 con 0x7f2d1c0f5640 2021-11-20T06:02:25.654 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.660+0000 7f2d21ac1700 1 --2- 172.21.15.178:0/4149773904 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f2d0805c2b0 0x7f2d0805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:25.655 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.661+0000 7f2d21ac1700 1 --2- 172.21.15.178:0/4149773904 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f2d0805c2b0 0x7f2d0805e770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f2d0c0041d0 tx=0x7f2d0c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:25.655 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.664+0000 7f2d12ffd700 1 -- 172.21.15.178:0/4149773904 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d1804aa20 con 0x7f2d1c0f5640 2021-11-20T06:02:25.865 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.879+0000 7f2d24526700 1 -- 172.21.15.178:0/4149773904 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f2d1c045ad0 con 0x7f2d1c0f5640 2021-11-20T06:02:25.869 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.884+0000 7f2d12ffd700 1 -- 172.21.15.178:0/4149773904 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f2d1804e040 con 0x7f2d1c0f5640 2021-11-20T06:02:25.870 INFO:teuthology.orchestra.run.smithi178.stdout:[client.1] 2021-11-20T06:02:25.871 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQBxj5hhjsmJNBAA1+1cfWgefZt8KncAPMf3vw== 2021-11-20T06:02:25.872 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.886+0000 7f2d24526700 1 -- 172.21.15.178:0/4149773904 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f2d0805c2b0 msgr2=0x7f2d0805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:25.873 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.886+0000 7f2d24526700 1 --2- 172.21.15.178:0/4149773904 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f2d0805c2b0 0x7f2d0805e770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f2d0c0041d0 tx=0x7f2d0c009040).stop 2021-11-20T06:02:25.873 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.886+0000 7f2d24526700 1 -- 172.21.15.178:0/4149773904 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d1c0f5640 msgr2=0x7f2d1c10c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:25.874 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.886+0000 7f2d24526700 1 --2- 172.21.15.178:0/4149773904 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d1c0f5640 0x7f2d1c10c0e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f2d18009f50 tx=0x7f2d18009b30).stop 2021-11-20T06:02:25.874 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.886+0000 7f2d24526700 1 -- 172.21.15.178:0/4149773904 shutdown_connections 2021-11-20T06:02:25.875 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.886+0000 7f2d24526700 1 --2- 172.21.15.178:0/4149773904 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f2d0805c2b0 0x7f2d0805e770 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:25.875 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.886+0000 7f2d24526700 1 --2- 172.21.15.178:0/4149773904 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d1c0f5640 0x7f2d1c10c0e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:25.876 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.886+0000 7f2d24526700 1 --2- 172.21.15.178:0/4149773904 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d1c0f68e0 0x7f2d1c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:25.876 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.886+0000 7f2d24526700 1 -- 172.21.15.178:0/4149773904 >> 172.21.15.178:0/4149773904 conn(0x7f2d1c0f0da0 msgr2=0x7f2d1c0fa330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:25.876 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.886+0000 7f2d24526700 1 -- 172.21.15.178:0/4149773904 shutdown_connections 2021-11-20T06:02:25.877 INFO:teuthology.orchestra.run.smithi178.stderr:2021-11-20T06:02:25.886+0000 7f2d24526700 1 -- 172.21.15.178:0/4149773904 wait complete. 2021-11-20T06:02:26.308 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2021-11-20T06:02:26.308 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2021-11-20T06:02:26.309 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-11-20T06:02:26.349 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph osd dump --format=json 2021-11-20T06:02:26.442 INFO:journalctl@ceph.osd.5.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[52933]: debug 2021-11-20T06:02:26.180+0000 7fbf12108700 -1 osd.5 0 waiting for initial osdmap 2021-11-20T06:02:26.443 INFO:journalctl@ceph.osd.5.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[52933]: debug 2021-11-20T06:02:26.188+0000 7fbf0e083700 -1 osd.5 34 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-20T06:02:26.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: cluster 2021-11-20T06:02:24.197957+0000 mgr.smithi139.dplyxn 2021-11-20T06:02:26.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: (mgr.14162) 144 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:26.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: audit 2021-11-20T06:02:25.176993+0000 mon.smithi139 2021-11-20T06:02:26.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: (mon.0) 409 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-11-20T06:02:26.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: cluster 2021-11-20T06:02:26.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: 2021-11-20T06:02:25.177056+0000 2021-11-20T06:02:26.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: mon.smithi139 (mon.0) 410 : cluster [DBG] 2021-11-20T06:02:26.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: osdmap e33: 6 total, 5 up, 6 in 2021-11-20T06:02:26.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: audit 2021-11-20T06:02:26.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: 2021-11-20T06:02:25.177173 2021-11-20T06:02:26.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: +0000 mon.smithi139 (mon.0) 411 : 2021-11-20T06:02:26.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T06:02:26.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: audit 2021-11 2021-11-20T06:02:26.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: -20T06:02:25.177652+0000 mon.smithi178 ( 2021-11-20T06:02:26.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: mon.1) 15 : audit [INF] 2021-11-20T06:02:26.447 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: from='osd.5 [v2:172.21.15.178:6816/3932451665,v1:172.21.15.178:6817/3932451665]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2021-11-20T06:02:26.447 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: audit 2021-11-20T06:02:26.447 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: 2021-11-20T 2021-11-20T06:02:26.447 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: 06:02:25.177906+0000 mon.smithi139 2021-11-20T06:02:26.447 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: (mon.0) 412 : audit 2021-11-20T06:02:26.448 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2021-11-20T06:02:26.448 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: audit 2021-11- 2021-11-20T06:02:26.448 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: 20T06:02:25.837279+0000 mon.smithi139 ( 2021-11-20T06:02:26.448 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: mon.0) 413 : audit [INF] 2021-11-20T06:02:26.449 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:26.449 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: audit 2021-11-20T06:02:26.449 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: 2021-11-20T06:02 2021-11-20T06:02:26.449 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: :25.880856+0000 mon.smithi178 (mon.1) 16 2021-11-20T06:02:26.449 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: : audit [INF] from='client.? 172.21.15.178:0/4149773904' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T06:02:26.450 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: audit 2021- 2021-11-20T06:02:26.450 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: 11-20T06:02:25.881263+0000 2021-11-20T06:02:26.450 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: mon.smithi139 (mon.0) 414 : audit 2021-11-20T06:02:26.450 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T06:02:26.451 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: audit 2021-11 2021-11-20T06:02:26.451 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: -20T06:02:25.884387+0000 mon.smithi139 2021-11-20T06:02:26.451 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: (mon.0) 415 : audit 2021-11-20T06:02:26.451 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:26 smithi178 conmon[36600]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T06:02:26.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:26 smithi139 conmon[33501]: cluster 2021-11-20T06:02:24.197957+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:02:26.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:26 smithi139 conmon[33501]: 144 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:26.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:26 smithi139 conmon[33501]: audit 2021-11-20T06:02:25.176993+0000 mon.smithi139 (mon.0) 409 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-11-20T06:02:26.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:26 smithi139 conmon[33501]: cluster 2021-11-20T06:02:25.177056+0000 mon.smithi139 (mon.0) 410 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2021-11-20T06:02:26.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:26 smithi139 conmon[33501]: audit 2021-11-20T06:02:25.177173+0000 mon.smithi139 (mon.0) 411 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T06:02:26.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:26 smithi139 conmon[33501]: audit 2021-11-20T06:02:25.177652+0000 mon.smithi178 (mon.1) 15 : audit [INF] from='osd.5 [v2:172.21.15.178:6816/3932451665,v1:172.21.15.178:6817/3932451665]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2021-11-20T06:02:26.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:26 smithi139 conmon[33501]: audit 2021-11-20T06:02:25.177906+0000 mon.smithi139 (mon.0) 412 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2021-11-20T06:02:26.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:26 smithi139 conmon[33501]: audit 2021-11-20T06:02:25.837279+0000 mon.smithi139 (mon.0) 413 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:26.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:26 smithi139 conmon[33501]: audit 2021-11-20T06:02:25.880856+0000 mon.smithi178 (mon.1) 16 : audit [INF] from='client.? 172.21.15.178:0/4149773904' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T06:02:26.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:26 smithi139 conmon[33501]: audit 2021-11-20T06:02:25.881263+0000 mon.smithi139 (mon.0) 414 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-20T06:02:26.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:26 smithi139 conmon[33501]: audit 2021-11-20T06:02:25.884387+0000 mon.smithi139 (mon.0) 415 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-20T06:02:26.782 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:02:27.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:27 smithi178 conmon[36600]: audit 2021-11-20T06:02:26.178162+0000 mon.smithi139 (mon.0) 416 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2021-11-20T06:02:27.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:27 smithi178 conmon[36600]: cluster 2021-11-20T06:02:26.178241+0000 mon.smithi139 (mon.0) 417 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2021-11-20T06:02:27.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:27 smithi178 conmon[36600]: audit 2021-11-20T06:02:26.179332+0000 mon.smithi139 (mon.0) 418 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T06:02:27.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:27 smithi178 conmon[36600]: audit 2021-11-20T06:02:26.181101+0000 mon.smithi139 (mon.0) 419 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T06:02:27.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:27 smithi178 conmon[36600]: cluster 2021-11-20T06:02:26.198203+0000 mgr.smithi139.dplyxn (mgr.14162) 145 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:27.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:27 smithi178 conmon[36600]: cephadm 2021-11-20T06:02:26.981796+0000 mgr.smithi139.dplyxn (mgr.14162) 146 : cephadm [INF] Detected new or changed devices on smithi178 2021-11-20T06:02:27.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:27 smithi178 conmon[36600]: audit 2021-11-20T06:02:26.986556+0000 mon.smithi139 (mon.0) 420 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:27.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:27 smithi178 conmon[36600]: audit 2021-11-20T06:02:27.181066+0000 mon.smithi139 (mon.0) 421 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T06:02:27.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:27 smithi139 conmon[33501]: audit 2021-11-20T06:02:26.178162+0000 2021-11-20T06:02:27.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:27 smithi139 conmon[33501]: mon.smithi139 (mon.0) 416 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2021-11-20T06:02:27.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:27 smithi139 conmon[33501]: cluster 2021-11-20T06:02:26.178241+0000 mon.smithi139 (mon.0) 417 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2021-11-20T06:02:27.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:27 smithi139 conmon[33501]: audit 2021-11-20T06:02:26.179332+0000 mon.smithi139 (mon.0) 418 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T06:02:27.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:27 smithi139 conmon[33501]: audit 2021-11-20T06:02:26.181101+0000 mon.smithi139 (mon.0) 419 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T06:02:27.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:27 smithi139 conmon[33501]: cluster 2021-11-20T06:02:26.198203+0000 mgr.smithi139.dplyxn (mgr.14162) 145 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-20T06:02:27.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:27 smithi139 conmon[33501]: cephadm 2021-11-20T06:02:26.981796+0000 mgr.smithi139.dplyxn (mgr.14162) 146 : cephadm [INF] Detected new or changed devices on smithi178 2021-11-20T06:02:27.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:27 smithi139 conmon[33501]: audit 2021-11-20T06:02:26.986556+0000 mon.smithi139 (mon.0) 420 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:27.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:27 smithi139 conmon[33501]: audit 2021-11-20T06:02:27.181066+0000 mon.smithi139 (mon.0) 421 2021-11-20T06:02:27.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:27 smithi139 conmon[33501]: : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T06:02:28.005 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.018+0000 7f45b0cb5700 1 -- 172.21.15.139:0/4129790383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45ac0f5520 msgr2=0x7f45ac0f5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:28.005 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.018+0000 7f45b0cb5700 1 --2- 172.21.15.139:0/4129790383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45ac0f5520 0x7f45ac0f5940 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f459c004660 tx=0x7f459c009b30).stop 2021-11-20T06:02:28.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.018+0000 7f45b0cb5700 1 -- 172.21.15.139:0/4129790383 shutdown_connections 2021-11-20T06:02:28.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.018+0000 7f45b0cb5700 1 --2- 172.21.15.139:0/4129790383 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45ac0f67c0 0x7f45ac0f6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:28.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.018+0000 7f45b0cb5700 1 --2- 172.21.15.139:0/4129790383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45ac0f5520 0x7f45ac0f5940 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:28.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.018+0000 7f45b0cb5700 1 -- 172.21.15.139:0/4129790383 >> 172.21.15.139:0/4129790383 conn(0x7f45ac0f0ca0 msgr2=0x7f45ac0f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:28.007 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.018+0000 7f45b0cb5700 1 -- 172.21.15.139:0/4129790383 shutdown_connections 2021-11-20T06:02:28.007 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.018+0000 7f45b0cb5700 1 -- 172.21.15.139:0/4129790383 wait complete. 2021-11-20T06:02:28.007 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.018+0000 7f45b0cb5700 1 Processor -- start 2021-11-20T06:02:28.007 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.019+0000 7f45b0cb5700 1 -- start start 2021-11-20T06:02:28.008 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.019+0000 7f45b0cb5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45ac0f5520 0x7f45ac105e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:28.008 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.019+0000 7f45b0cb5700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45ac0f67c0 0x7f45ac106320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:28.009 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.019+0000 7f45b0cb5700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac106920 con 0x7f45ac0f5520 2021-11-20T06:02:28.009 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.019+0000 7f45b0cb5700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac107130 con 0x7f45ac0f67c0 2021-11-20T06:02:28.009 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.019+0000 7f45ab7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45ac0f5520 0x7f45ac105e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:28.009 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.019+0000 7f45ab7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45ac0f5520 0x7f45ac105e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57108/0 (socket says 172.21.15.139:57108) 2021-11-20T06:02:28.010 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.019+0000 7f45ab7fe700 1 -- 172.21.15.139:0/1884203884 learned_addr learned my addr 172.21.15.139:0/1884203884 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:28.010 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.019+0000 7f45ab7fe700 1 -- 172.21.15.139:0/1884203884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45ac0f67c0 msgr2=0x7f45ac106320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:28.010 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.019+0000 7f45aaffd700 1 --2- 172.21.15.139:0/1884203884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45ac0f67c0 0x7f45ac106320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:28.013 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.019+0000 7f45ab7fe700 1 --2- 172.21.15.139:0/1884203884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45ac0f67c0 0x7f45ac106320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:28.013 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.019+0000 7f45ab7fe700 1 -- 172.21.15.139:0/1884203884 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c005040 con 0x7f45ac0f5520 2021-11-20T06:02:28.013 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.019+0000 7f45aaffd700 1 --2- 172.21.15.139:0/1884203884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45ac0f67c0 0x7f45ac106320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:02:28.014 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.020+0000 7f45ab7fe700 1 --2- 172.21.15.139:0/1884203884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45ac0f5520 0x7f45ac105e30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f459c004e30 tx=0x7f459c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:28.014 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.020+0000 7f4593fff700 1 -- 172.21.15.139:0/1884203884 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f459c018070 con 0x7f45ac0f5520 2021-11-20T06:02:28.014 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.020+0000 7f4593fff700 1 -- 172.21.15.139:0/1884203884 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f459c013410 con 0x7f45ac0f5520 2021-11-20T06:02:28.014 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.020+0000 7f45b0cb5700 1 -- 172.21.15.139:0/1884203884 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ac107350 con 0x7f45ac0f5520 2021-11-20T06:02:28.014 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.020+0000 7f4593fff700 1 -- 172.21.15.139:0/1884203884 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f459c01c3c0 con 0x7f45ac0f5520 2021-11-20T06:02:28.015 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.020+0000 7f45b0cb5700 1 -- 172.21.15.139:0/1884203884 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ac1078b0 con 0x7f45ac0f5520 2021-11-20T06:02:28.015 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.021+0000 7f45b0cb5700 1 -- 172.21.15.139:0/1884203884 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ac045ad0 con 0x7f45ac0f5520 2021-11-20T06:02:28.015 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.022+0000 7f4593fff700 1 -- 172.21.15.139:0/1884203884 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f459c01d420 con 0x7f45ac0f5520 2021-11-20T06:02:28.015 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.022+0000 7f4593fff700 1 --2- 172.21.15.139:0/1884203884 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f459405c2b0 0x7f459405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:28.016 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.022+0000 7f4593fff700 1 -- 172.21.15.139:0/1884203884 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f459c077310 con 0x7f45ac0f5520 2021-11-20T06:02:28.016 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.022+0000 7f45aaffd700 1 --2- 172.21.15.139:0/1884203884 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f459405c2b0 0x7f459405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:28.016 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.022+0000 7f45aaffd700 1 --2- 172.21.15.139:0/1884203884 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f459405c2b0 0x7f459405e770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f45a0009880 tx=0x7f45a000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:28.016 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.026+0000 7f4593fff700 1 -- 172.21.15.139:0/1884203884 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f459c04a310 con 0x7f45ac0f5520 2021-11-20T06:02:28.186 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.199+0000 7f45b0cb5700 1 -- 172.21.15.139:0/1884203884 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45ac107e60 con 0x7f45ac0f5520 2021-11-20T06:02:28.187 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.200+0000 7f4593fff700 1 -- 172.21.15.139:0/1884203884 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+10925 (secure 0 0 0) 0x7f459c04d930 con 0x7f45ac0f5520 2021-11-20T06:02:28.187 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:02:28.187 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":35,"fsid":"d819f2cc-49c6-11ec-8c2c-001a4aab830c","created":"2021-11-20T05:58:33.540072+0000","modified":"2021-11-20T06:02:27.182023+0000","last_up_change":"2021-11-20T06:02:27.182023+0000","last_in_change":"2021-11-20T06:02:11.372444+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T06:01:48.405544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"8fece5a6-7a49-451c-beb8-f90723d42033","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6803","nonce":22698595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6805","nonce":22698595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6809","nonce":22698595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6807","nonce":22698595}]},"public_addr":"172.21.15.139:6803/22698595","cluster_addr":"172.21.15.139:6805/22698595","heartbeat_back_addr":"172.21.15.139:6809/22698595","heartbeat_front_addr":"172.21.15.139:6807/22698595","state":["exists","up"]},{"osd":1,"uuid":"ce18ef72-7c6c-43f7-8a0c-6904f7c9e606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6811","nonce":2530906188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6813","nonce":2530906188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6817","nonce":2530906188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6815","nonce":2530906188}]},"public_addr":"172.21.15.139:6811/2530906188","cluster_addr":"172.21.15.139:6813/2530906188","heartbeat_back_addr":"172.21.15.139:6817/2530906188","heartbeat_front_addr":"172.21.15.139:6815/2530906188","state":["exists","up"]},{"osd":2,"uuid":"80cfe585-e283-45a4-9dd7-f328ec00d21c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6819","nonce":1147042903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6821","nonce":1147042903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6825","nonce":1147042903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6823","nonce":1147042903}]},"public_addr":"172.21.15.139:6819/1147042903","cluster_addr":"172.21.15.139:6821/1147042903","heartbeat_back_addr":"172.21.15.139:6825/1147042903","heartbeat_front_addr":"172.21.15.139:6823/1147042903","state":["exists","up"]},{"osd":3,"uuid":"793e383f-4d5a-41cb-b5b3-b4e4e07d6b8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6801","nonce":927121970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6803","nonce":927121970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6807","nonce":927121970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6805","nonce":927121970}]},"public_addr":"172.21.15.178:6801/927121970","cluster_addr":"172.21.15.178:6803/927121970","heartbeat_back_addr":"172.21.15.178:6807/927121970","heartbeat_front_addr":"172.21.15.178:6805/927121970","state":["exists","up"]},{"osd":4,"uuid":"3e453fbf-8ef9-4227-b3b8-cd6631bff6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6809","nonce":2601954583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6811","nonce":2601954583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6815","nonce":2601954583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6813","nonce":2601954583}]},"public_addr":"172.21.15.178:6809/2601954583","cluster_addr":"172.21.15.178:6811/2601954583","heartbeat_back_addr":"172.21.15.178:6815/2601954583","heartbeat_front_addr":"172.21.15.178:6813/2601954583","state":["exists","up"]},{"osd":5,"uuid":"b7d0eee4-3cb1-43dc-9c3f-f95593cf47cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6817","nonce":3932451665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6819","nonce":3932451665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6823","nonce":3932451665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6821","nonce":3932451665}]},"public_addr":"172.21.15.178:6817/3932451665","cluster_addr":"172.21.15.178:6819/3932451665","heartbeat_back_addr":"172.21.15.178:6823/3932451665","heartbeat_front_addr":"172.21.15.178:6821/3932451665","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:01.290566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:19.499151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:37.169640+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:52.838242+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:02:09.060491+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.139:0/3013495710":"2021-11-21T05:59:26.112534+0000","172.21.15.139:0/1461722096":"2021-11-21T05:59:13.525765+0000","172.21.15.139:0/1750602715":"2021-11-21T05:59:26.112534+0000","172.21.15.139:6800/3637001875":"2021-11-21T05:59:26.112534+0000","172.21.15.139:6801/1220659339":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6800/1220659339":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6801/3637001875":"2021-11-21T05:59:26.112534+0000","172.21.15.139:0/2668621147":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6800/4049861874":"2021-11-21T05:58:51.653389+0000","172.21.15.139:0/2863654682":"2021-11-21T05:58:51.653389+0000","172.21.15.139:0/2447016509":"2021-11-21T05:58:51.653389+0000","172.21.15.139:6801/4049861874":"2021-11-21T05:58:51.653389+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T06:02:28.189 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.202+0000 7f45b0cb5700 1 -- 172.21.15.139:0/1884203884 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f459405c2b0 msgr2=0x7f459405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:28.189 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.202+0000 7f45b0cb5700 1 --2- 172.21.15.139:0/1884203884 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f459405c2b0 0x7f459405e770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f45a0009880 tx=0x7f45a000b040).stop 2021-11-20T06:02:28.189 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.202+0000 7f45b0cb5700 1 -- 172.21.15.139:0/1884203884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45ac0f5520 msgr2=0x7f45ac105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:28.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.202+0000 7f45b0cb5700 1 --2- 172.21.15.139:0/1884203884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45ac0f5520 0x7f45ac105e30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f459c004e30 tx=0x7f459c009b30).stop 2021-11-20T06:02:28.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.202+0000 7f45b0cb5700 1 -- 172.21.15.139:0/1884203884 shutdown_connections 2021-11-20T06:02:28.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.202+0000 7f45b0cb5700 1 --2- 172.21.15.139:0/1884203884 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f459405c2b0 0x7f459405e770 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:28.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.202+0000 7f45b0cb5700 1 --2- 172.21.15.139:0/1884203884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f45ac0f5520 0x7f45ac105e30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:28.191 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.202+0000 7f45b0cb5700 1 --2- 172.21.15.139:0/1884203884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45ac0f67c0 0x7f45ac106320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:28.191 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.202+0000 7f45b0cb5700 1 -- 172.21.15.139:0/1884203884 >> 172.21.15.139:0/1884203884 conn(0x7f45ac0f0ca0 msgr2=0x7f45ac0fa210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:28.191 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.202+0000 7f45b0cb5700 1 -- 172.21.15.139:0/1884203884 shutdown_connections 2021-11-20T06:02:28.191 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:28.202+0000 7f45b0cb5700 1 -- 172.21.15.139:0/1884203884 wait complete. 2021-11-20T06:02:28.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:28 smithi178 conmon[36600]: cluster 2021-11-20T06:02:25.766578+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T06:02:28.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:28 smithi178 conmon[36600]: cluster 2021-11-20T06:02:25.766676+0000 2021-11-20T06:02:28.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:28 smithi178 conmon[36600]: osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T06:02:28.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:28 smithi178 conmon[36600]: cluster 2021-11-20T06:02:27.185834+0000 mon.smithi139 (mon.0) 422 : cluster [INF] osd.5 [v2:172.21.15.178:6816/3932451665,v1:172.21.15.178:6817/3932451665] boot 2021-11-20T06:02:28.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:28 smithi178 conmon[36600]: cluster 2021-11-20T06:02:27.185882+0000 2021-11-20T06:02:28.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:28 smithi178 conmon[36600]: mon.smithi139 (mon.0) 423 : cluster [DBG] osdmap e35: 6 total, 6 up, 6 in 2021-11-20T06:02:28.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:28 smithi178 conmon[36600]: audit 2021-11-20T06:02 2021-11-20T06:02:28.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:28 smithi178 conmon[36600]: :27.186271+0000 mon.smithi139 (mon.0) 424 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T06:02:28.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:28 smithi178 conmon[36600]: 2021-11-20T06:02:28.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:28 smithi178 conmon[36600]: audit 2021-11 2021-11-20T06:02:28.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:28 smithi178 conmon[36600]: -20T06:02:27.342588+0000 mon.smithi139 (mon.0) 425 2021-11-20T06:02:28.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:28 smithi178 conmon[36600]: : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:28.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:28 smithi178 conmon[36600]: audit 2021-11-20T 2021-11-20T06:02:28.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:28 smithi178 conmon[36600]: 06:02:27.345074+0000 mon.smithi139 (mon.0 2021-11-20T06:02:28.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:28 smithi178 conmon[36600]: ) 426 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:02:28.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:28 smithi139 conmon[33501]: cluster 2021-11-20T06:02:25.766578+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-20T06:02:28.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:28 smithi139 conmon[33501]: cluster 2021-11-20T06:02:25.766676+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-20T06:02:28.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:28 smithi139 conmon[33501]: cluster 2021-11-20T06:02:27.185834+0000 mon.smithi139 (mon.0) 422 : cluster [INF] osd.5 [v2:172.21.15.178:6816/3932451665,v1:172.21.15.178:6817/3932451665] boot 2021-11-20T06:02:28.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:28 smithi139 conmon[33501]: cluster 2021-11-20T06:02:27.185882+0000 mon.smithi139 (mon.0) 423 : cluster [DBG] osdmap e35: 6 total, 6 up, 6 in 2021-11-20T06:02:28.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:28 smithi139 conmon[33501]: audit 2021-11-20T06:02:27.186271+0000 mon.smithi139 (mon.0) 424 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-20T06:02:28.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:28 smithi139 conmon[33501]: audit 2021-11-20T06:02:27.342588+0000 mon.smithi139 (mon.0) 425 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:28.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:28 smithi139 conmon[33501]: audit 2021-11-20T06:02:27.345074+0000 mon.smithi139 2021-11-20T06:02:28.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:28 smithi139 conmon[33501]: (mon.0) 426 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:02:28.691 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T06:01:48.405544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2021-11-20T06:02:28.692 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph osd pool get device_health_metrics pg_num 2021-11-20T06:02:29.136 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:02:29.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:29 smithi178 conmon[36600]: cluster 2021-11-20T06:02:28.198481+0000 mgr.smithi139.dplyxn (mgr.14162) 147 : cluster 2021-11-20T06:02:29.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:29 smithi178 conmon[36600]: [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:29.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:29 smithi178 conmon[36600]: audit 2021-11-20T06:02:28.201672+0000 mon.smithi139 (mon.0) 427 : audit [DBG] from='client.? 172.21.15.139:0/1884203884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T06:02:29.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:29 smithi178 conmon[36600]: cluster 2021-11-20T06:02:28.345314+0000 2021-11-20T06:02:29.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:29 smithi178 conmon[36600]: mon.smithi139 (mon.0) 428 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2021-11-20T06:02:29.524 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:29 smithi139 conmon[33501]: cluster 2021-11-20T06:02:28.198481+0000 mgr.smithi139.dplyxn 2021-11-20T06:02:29.524 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:29 smithi139 conmon[33501]: (mgr.14162) 147 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:29.524 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:29 smithi139 conmon[33501]: audit 2021-11-20T06:02:28.201672+0000 mon.smithi139 (mon.0) 427 : audit [DBG] from='client.? 172.21.15.139:0/1884203884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T06:02:29.525 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:29 smithi139 conmon[33501]: cluster 2021-11-20T06:02:28.345314+0000 mon.smithi139 (mon.0) 428 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2021-11-20T06:02:30.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.498+0000 7fb098fdc700 1 -- 172.21.15.139:0/2927018296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0940f7a60 msgr2=0x7fb0940f7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:30.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.498+0000 7fb098fdc700 1 --2- 172.21.15.139:0/2927018296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0940f7a60 0x7fb0940f7ec0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fb088004660 tx=0x7fb088009b30).stop 2021-11-20T06:02:30.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.498+0000 7fb098fdc700 1 -- 172.21.15.139:0/2927018296 shutdown_connections 2021-11-20T06:02:30.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.498+0000 7fb098fdc700 1 --2- 172.21.15.139:0/2927018296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0940f7a60 0x7fb0940f7ec0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:30.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.498+0000 7fb098fdc700 1 --2- 172.21.15.139:0/2927018296 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0940f67c0 0x7fb0940f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:30.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.498+0000 7fb098fdc700 1 -- 172.21.15.139:0/2927018296 >> 172.21.15.139:0/2927018296 conn(0x7fb0940f1f20 msgr2=0x7fb0940f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:30.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.498+0000 7fb098fdc700 1 -- 172.21.15.139:0/2927018296 shutdown_connections 2021-11-20T06:02:30.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.498+0000 7fb098fdc700 1 -- 172.21.15.139:0/2927018296 wait complete. 2021-11-20T06:02:30.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.499+0000 7fb098fdc700 1 Processor -- start 2021-11-20T06:02:30.490 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.499+0000 7fb098fdc700 1 -- start start 2021-11-20T06:02:30.490 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.499+0000 7fb098fdc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0940f67c0 0x7fb0941070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:30.491 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.499+0000 7fb098fdc700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0940f7a60 0x7fb094107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:30.491 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.499+0000 7fb098fdc700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb094107b40 con 0x7fb0940f67c0 2021-11-20T06:02:30.491 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.499+0000 7fb098fdc700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb094107c80 con 0x7fb0940f7a60 2021-11-20T06:02:30.491 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.499+0000 7fb09259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0940f67c0 0x7fb0941070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:30.492 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.499+0000 7fb09259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0940f67c0 0x7fb0941070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57128/0 (socket says 172.21.15.139:57128) 2021-11-20T06:02:30.492 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.499+0000 7fb09259c700 1 -- 172.21.15.139:0/1566534614 learned_addr learned my addr 172.21.15.139:0/1566534614 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:30.492 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.499+0000 7fb091d9b700 1 --2- 172.21.15.139:0/1566534614 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0940f7a60 0x7fb094107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:30.492 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.499+0000 7fb09259c700 1 -- 172.21.15.139:0/1566534614 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0940f7a60 msgr2=0x7fb094107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:30.493 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.499+0000 7fb09259c700 1 --2- 172.21.15.139:0/1566534614 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0940f7a60 0x7fb094107590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:30.493 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.499+0000 7fb09259c700 1 -- 172.21.15.139:0/1566534614 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb088005040 con 0x7fb0940f67c0 2021-11-20T06:02:30.493 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.499+0000 7fb09259c700 1 --2- 172.21.15.139:0/1566534614 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0940f67c0 0x7fb0941070a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fb084002700 tx=0x7fb084009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:30.494 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.500+0000 7fb082ffd700 1 -- 172.21.15.139:0/1566534614 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0840079c0 con 0x7fb0940f67c0 2021-11-20T06:02:30.494 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.500+0000 7fb082ffd700 1 -- 172.21.15.139:0/1566534614 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb08400f040 con 0x7fb0940f67c0 2021-11-20T06:02:30.494 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.500+0000 7fb098fdc700 1 -- 172.21.15.139:0/1566534614 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb094108620 con 0x7fb0940f67c0 2021-11-20T06:02:30.494 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.500+0000 7fb082ffd700 1 -- 172.21.15.139:0/1566534614 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb084014860 con 0x7fb0940f67c0 2021-11-20T06:02:30.495 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.500+0000 7fb098fdc700 1 -- 172.21.15.139:0/1566534614 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb094108be0 con 0x7fb0940f67c0 2021-11-20T06:02:30.495 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.500+0000 7fb098fdc700 1 -- 172.21.15.139:0/1566534614 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb094045ad0 con 0x7fb0940f67c0 2021-11-20T06:02:30.495 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.501+0000 7fb082ffd700 1 -- 172.21.15.139:0/1566534614 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb08401b060 con 0x7fb0940f67c0 2021-11-20T06:02:30.497 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.501+0000 7fb082ffd700 1 --2- 172.21.15.139:0/1566534614 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb07c05c2a0 0x7fb07c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:30.497 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.501+0000 7fb082ffd700 1 -- 172.21.15.139:0/1566534614 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fb084075a70 con 0x7fb0940f67c0 2021-11-20T06:02:30.498 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.501+0000 7fb091d9b700 1 --2- 172.21.15.139:0/1566534614 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb07c05c2a0 0x7fb07c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:30.498 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.502+0000 7fb091d9b700 1 --2- 172.21.15.139:0/1566534614 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb07c05c2a0 0x7fb07c05e760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fb088009f50 tx=0x7fb088009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:30.498 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.505+0000 7fb082ffd700 1 -- 172.21.15.139:0/1566534614 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb084048a70 con 0x7fb0940f67c0 2021-11-20T06:02:30.646 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.659+0000 7fb098fdc700 1 -- 172.21.15.139:0/1566534614 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fb094027a00 con 0x7fb0940f67c0 2021-11-20T06:02:30.646 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.659+0000 7fb082ffd700 1 -- 172.21.15.139:0/1566534614 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v36) v1 ==== 110+0+10 (secure 0 0 0) 0x7fb084075e60 con 0x7fb0940f67c0 2021-11-20T06:02:30.647 INFO:teuthology.orchestra.run.smithi139.stdout:pg_num: 1 2021-11-20T06:02:30.648 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.661+0000 7fb098fdc700 1 -- 172.21.15.139:0/1566534614 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb07c05c2a0 msgr2=0x7fb07c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:30.648 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.661+0000 7fb098fdc700 1 --2- 172.21.15.139:0/1566534614 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb07c05c2a0 0x7fb07c05e760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fb088009f50 tx=0x7fb088009b30).stop 2021-11-20T06:02:30.648 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.661+0000 7fb098fdc700 1 -- 172.21.15.139:0/1566534614 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0940f67c0 msgr2=0x7fb0941070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:30.649 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.661+0000 7fb098fdc700 1 --2- 172.21.15.139:0/1566534614 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0940f67c0 0x7fb0941070a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fb084002700 tx=0x7fb084009d80).stop 2021-11-20T06:02:30.649 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.661+0000 7fb098fdc700 1 -- 172.21.15.139:0/1566534614 shutdown_connections 2021-11-20T06:02:30.649 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.661+0000 7fb098fdc700 1 --2- 172.21.15.139:0/1566534614 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb07c05c2a0 0x7fb07c05e760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:30.649 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.661+0000 7fb098fdc700 1 --2- 172.21.15.139:0/1566534614 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb0940f67c0 0x7fb0941070a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:30.650 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.661+0000 7fb098fdc700 1 --2- 172.21.15.139:0/1566534614 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0940f7a60 0x7fb094107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:30.650 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.661+0000 7fb098fdc700 1 -- 172.21.15.139:0/1566534614 >> 172.21.15.139:0/1566534614 conn(0x7fb0940f1f20 msgr2=0x7fb0940fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:30.650 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.661+0000 7fb098fdc700 1 -- 172.21.15.139:0/1566534614 shutdown_connections 2021-11-20T06:02:30.651 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:30.661+0000 7fb098fdc700 1 -- 172.21.15.139:0/1566534614 wait complete. 2021-11-20T06:02:31.089 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-11-20T06:02:31.090 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2021-11-20T06:02:31.090 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph mgr dump --format=json 2021-11-20T06:02:31.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:31 smithi139 conmon[33501]: cluster 2021-11-20T06:02:30.198849+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:02:31.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:31 smithi139 conmon[33501]: mgr.14162) 148 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:31.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:31 smithi139 conmon[33501]: audit 2021-11-20T06:02:30.661348+0000 mon.smithi139 (mon.0) 429 : audit [DBG] from='client.? 172.21.15.139:0/1566534614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T06:02:31.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:31 smithi178 conmon[36600]: cluster 2021-11-20T06:02:30.198849+0000 mgr.smithi139.dplyxn (mgr.14162) 148 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:31.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:31 smithi178 conmon[36600]: audit 2021- 2021-11-20T06:02:31.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:31 smithi178 conmon[36600]: 11-20T06:02:30.661348+0000 mon.smithi139 (mon.0) 429 : audit [DBG] from='client.? 172.21.15.139:0/1566534614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T06:02:31.799 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:02:32.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:32 smithi178 conmon[36600]: audit 2021-11-20T06:02:31.250917+0000 mon.smithi139 (mon.0) 430 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:02:32.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:32 smithi178 conmon[36600]: audit 2021-11-20T06:02: 2021-11-20T06:02:32.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:32 smithi178 conmon[36600]: 31.251071+0000 mon.smithi139 (mon.0) 431 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:02:32.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:32 smithi139 conmon[33501]: audit 2021-11-20T06:02:31.250917+0000 mon.smithi139 (mon.0 2021-11-20T06:02:32.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:32 smithi139 conmon[33501]: ) 430 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:02:32.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:32 smithi139 conmon[33501]: audit 2021-11-20T06:02:31.251071+0000 mon.smithi139 (mon.0) 431 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:02:33.357 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.368+0000 7f7b45047700 1 -- 172.21.15.139:0/1574860503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b400f56e0 msgr2=0x7f7b400f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:33.357 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.368+0000 7f7b45047700 1 --2- 172.21.15.139:0/1574860503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b400f56e0 0x7f7b400f5b00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f7b38004660 tx=0x7f7b38009b30).stop 2021-11-20T06:02:33.357 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.368+0000 7f7b45047700 1 -- 172.21.15.139:0/1574860503 shutdown_connections 2021-11-20T06:02:33.358 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.368+0000 7f7b45047700 1 --2- 172.21.15.139:0/1574860503 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b400f6980 0x7f7b400f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:02:33.358 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.368+0000 7f7b45047700 1 --2- 172.21.15.139:0/1574860503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b400f56e0 0x7f7b400f5b00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:33.358 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.368+0000 7f7b45047700 1 -- 172.21.15.139:0/1574860503 >> 172.21.15.139:0/1574860503 conn(0x7f7b400f0e20 msgr2=0x7f7b400f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:33.358 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.368+0000 7f7b45047700 1 -- 172.21.15.139:0/1574860503 shutdown_connections 2021-11-20T06:02:33.359 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.368+0000 7f7b45047700 1 -- 172.21.15.139:0/1574860503 wait complete. 2021-11-20T06:02:33.359 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.368+0000 7f7b45047700 1 Processor -- start 2021-11-20T06:02:33.360 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.368+0000 7f7b45047700 1 -- start start 2021-11-20T06:02:33.360 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.368+0000 7f7b45047700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b400f56e0 0x7f7b400fa060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:33.360 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b45047700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b400f6980 0x7f7b400f84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:33.360 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b45047700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b400f8ad0 con 0x7f7b400f56e0 2021-11-20T06:02:33.361 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b45047700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b400f8c10 con 0x7f7b400f6980 2021-11-20T06:02:33.361 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b3ed9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b400f56e0 0x7f7b400fa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:33.361 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b3ed9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b400f56e0 0x7f7b400fa060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57138/0 (socket says 172.21.15.139:57138) 2021-11-20T06:02:33.361 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b3ed9d700 1 -- 172.21.15.139:0/2728026763 learned_addr learned my addr 172.21.15.139:0/2728026763 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:33.362 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b3e59c700 1 --2- 172.21.15.139:0/2728026763 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b400f6980 0x7f7b400f84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:33.363 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b3ed9d700 1 -- 172.21.15.139:0/2728026763 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b400f6980 msgr2=0x7f7b400f84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:33.364 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b3ed9d700 1 --2- 172.21.15.139:0/2728026763 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b400f6980 0x7f7b400f84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:33.365 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b3ed9d700 1 -- 172.21.15.139:0/2728026763 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b38005040 con 0x7f7b400f56e0 2021-11-20T06:02:33.365 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b3ed9d700 1 --2- 172.21.15.139:0/2728026763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b400f56e0 0x7f7b400fa060 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7b38009f50 tx=0x7f7b38009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:33.366 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b377fe700 1 -- 172.21.15.139:0/2728026763 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b38018070 con 0x7f7b400f56e0 2021-11-20T06:02:33.366 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b377fe700 1 -- 172.21.15.139:0/2728026763 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b38013410 con 0x7f7b400f56e0 2021-11-20T06:02:33.366 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b45047700 1 -- 172.21.15.139:0/2728026763 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b400f8e90 con 0x7f7b400f56e0 2021-11-20T06:02:33.367 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b377fe700 1 -- 172.21.15.139:0/2728026763 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b3801c3c0 con 0x7f7b400f56e0 2021-11-20T06:02:33.367 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.369+0000 7f7b45047700 1 -- 172.21.15.139:0/2728026763 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4010f100 con 0x7f7b400f56e0 2021-11-20T06:02:33.368 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.370+0000 7f7b45047700 1 -- 172.21.15.139:0/2728026763 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b40045ad0 con 0x7f7b400f56e0 2021-11-20T06:02:33.368 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.371+0000 7f7b377fe700 1 -- 172.21.15.139:0/2728026763 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7b3801d420 con 0x7f7b400f56e0 2021-11-20T06:02:33.370 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.371+0000 7f7b377fe700 1 --2- 172.21.15.139:0/2728026763 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7b2c05c2a0 0x7f7b2c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:33.370 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.371+0000 7f7b377fe700 1 -- 172.21.15.139:0/2728026763 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7b380778e0 con 0x7f7b400f56e0 2021-11-20T06:02:33.370 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.371+0000 7f7b3e59c700 1 --2- 172.21.15.139:0/2728026763 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7b2c05c2a0 0x7f7b2c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:33.371 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.371+0000 7f7b3e59c700 1 --2- 172.21.15.139:0/2728026763 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7b2c05c2a0 0x7f7b2c05e760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f7b28006fd0 tx=0x7f7b28008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:33.371 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.375+0000 7f7b377fe700 1 -- 172.21.15.139:0/2728026763 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b3804a8e0 con 0x7f7b400f56e0 2021-11-20T06:02:33.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.564+0000 7f7b45047700 1 -- 172.21.15.139:0/2728026763 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f7b4010f7b0 con 0x7f7b400f56e0 2021-11-20T06:02:33.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.567+0000 7f7b377fe700 1 -- 172.21.15.139:0/2728026763 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+143505 (secure 0 0 0) 0x7f7b3804df00 con 0x7f7b400f56e0 2021-11-20T06:02:33.558 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:02:33.561 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.569+0000 7f7b45047700 1 -- 172.21.15.139:0/2728026763 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7b2c05c2a0 msgr2=0x7f7b2c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:33.562 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.569+0000 7f7b45047700 1 --2- 172.21.15.139:0/2728026763 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7b2c05c2a0 0x7f7b2c05e760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f7b28006fd0 tx=0x7f7b28008040).stop 2021-11-20T06:02:33.562 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.569+0000 7f7b45047700 1 -- 172.21.15.139:0/2728026763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b400f56e0 msgr2=0x7f7b400fa060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:33.563 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.569+0000 7f7b45047700 1 --2- 172.21.15.139:0/2728026763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b400f56e0 0x7f7b400fa060 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7b38009f50 tx=0x7f7b38009b30).stop 2021-11-20T06:02:33.563 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.569+0000 7f7b45047700 1 -- 172.21.15.139:0/2728026763 shutdown_connections 2021-11-20T06:02:33.564 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.569+0000 7f7b45047700 1 --2- 172.21.15.139:0/2728026763 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7b2c05c2a0 0x7f7b2c05e760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:33.564 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.569+0000 7f7b45047700 1 --2- 172.21.15.139:0/2728026763 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b400f56e0 0x7f7b400fa060 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:33.565 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.569+0000 7f7b45047700 1 --2- 172.21.15.139:0/2728026763 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b400f6980 0x7f7b400f84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:33.566 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.569+0000 7f7b45047700 1 -- 172.21.15.139:0/2728026763 >> 172.21.15.139:0/2728026763 conn(0x7f7b400f0e20 msgr2=0x7f7b4005fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:33.566 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.570+0000 7f7b45047700 1 -- 172.21.15.139:0/2728026763 shutdown_connections 2021-11-20T06:02:33.566 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:33.570+0000 7f7b45047700 1 -- 172.21.15.139:0/2728026763 wait complete. 2021-11-20T06:02:34.114 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":17,"active_gid":14162,"active_name":"smithi139.dplyxn","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":2363962993},{"type":"v1","addr":"172.21.15.139:6801","nonce":2363962993}]},"active_addr":"172.21.15.139:6801/2363962993","active_change":"2021-11-20T05:59:26.112818+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":14198,"name":"smithi178.zxxdkz","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://smithi139.front.sepia.ceph.com:8443/","prometheus":"http://smithi139.front.sepia.ceph.com:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":4,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.139:0","nonce":3987749478}]},{"addrvec":[{"type":"v2","addr":"172.21.15.139:0","nonce":1437508174}]}]}} 2021-11-20T06:02:34.120 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2021-11-20T06:02:34.120 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2021-11-20T06:02:34.121 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph osd dump --format=json 2021-11-20T06:02:34.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:34 smithi178 conmon[36600]: cluster 2021-11-20T06:02:32.199257+0000 mgr.smithi139.dplyxn (mgr.14162) 149 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:34.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:34 smithi178 conmon[36600]: audit 2021-11-20T06:02:33.456677+0000 2021-11-20T06:02:34.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:34 smithi178 conmon[36600]: mon.smithi139 (mon.0) 432 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:34.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:34 smithi178 conmon[36600]: audit 2021-11-20T06:02:33.459588+0000 mon.smithi139 (mon.0) 433 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:34.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:34 smithi178 conmon[36600]: audit 2021-11-20T06:02:33.566626+0000 mon.smithi139 (mon.0) 434 2021-11-20T06:02:34.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:34 smithi178 conmon[36600]: : audit [DBG] from='client.? 172.21.15.139:0/2728026763' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-11-20T06:02:34.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:34 smithi139 conmon[33501]: cluster 2021-11-20T06:02:32.199257+0000 mgr.smithi139.dplyxn (mgr.14162) 149 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:34.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:34 smithi139 conmon[33501]: audit 2021-11-20T06:02:33.456677+0000 mon.smithi139 (mon.0) 432 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:34.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:34 smithi139 conmon[33501]: audit 2021-11-20T06:02:33.459588+0000 mon.smithi139 (mon.0) 433 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:02:34.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:34 smithi139 conmon[33501]: audit 2021-11-20T06:02:33.566626+0000 mon.smithi139 (mon.0) 434 : audit [DBG] from='client.? 172.21.15.139:0/2728026763' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-11-20T06:02:34.559 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:02:35.791 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.803+0000 7f6371053700 1 -- 172.21.15.139:0/1617721219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636c0f5440 msgr2=0x7f636c0f7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:35.791 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.803+0000 7f6371053700 1 --2- 172.21.15.139:0/1617721219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636c0f5440 0x7f636c0f7bc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f63600045a0 tx=0x7f6360009b30).stop 2021-11-20T06:02:35.791 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.803+0000 7f6371053700 1 -- 172.21.15.139:0/1617721219 shutdown_connections 2021-11-20T06:02:35.792 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.803+0000 7f6371053700 1 --2- 172.21.15.139:0/1617721219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636c0f5440 0x7f636c0f7bc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:35.792 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.803+0000 7f6371053700 1 --2- 172.21.15.139:0/1617721219 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f636c0f4aa0 0x7f636c0f4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:35.792 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.803+0000 7f6371053700 1 -- 172.21.15.139:0/1617721219 >> 172.21.15.139:0/1617721219 conn(0x7f636c0f05e0 msgr2=0x7f636c0f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:35.792 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.803+0000 7f6371053700 1 -- 172.21.15.139:0/1617721219 shutdown_connections 2021-11-20T06:02:35.793 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.803+0000 7f6371053700 1 -- 172.21.15.139:0/1617721219 wait complete. 2021-11-20T06:02:35.793 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.804+0000 7f6371053700 1 Processor -- start 2021-11-20T06:02:35.793 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.804+0000 7f6371053700 1 -- start start 2021-11-20T06:02:35.793 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.804+0000 7f6371053700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636c0f4aa0 0x7f636c107150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:35.793 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.804+0000 7f6371053700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f636c0f5440 0x7f636c107640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:35.794 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.804+0000 7f6371053700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c107bf0 con 0x7f636c0f4aa0 2021-11-20T06:02:35.794 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.804+0000 7f6371053700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c107d30 con 0x7f636c0f5440 2021-11-20T06:02:35.794 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.804+0000 7f636ad9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636c0f4aa0 0x7f636c107150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:35.794 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.804+0000 7f636ad9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636c0f4aa0 0x7f636c107150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57148/0 (socket says 172.21.15.139:57148) 2021-11-20T06:02:35.795 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.804+0000 7f636ad9d700 1 -- 172.21.15.139:0/3420036005 learned_addr learned my addr 172.21.15.139:0/3420036005 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:35.795 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.804+0000 7f636a59c700 1 --2- 172.21.15.139:0/3420036005 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f636c0f5440 0x7f636c107640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:35.795 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.804+0000 7f636ad9d700 1 -- 172.21.15.139:0/3420036005 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f636c0f5440 msgr2=0x7f636c107640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:35.795 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.804+0000 7f636ad9d700 1 --2- 172.21.15.139:0/3420036005 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f636c0f5440 0x7f636c107640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:35.795 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.804+0000 7f636ad9d700 1 -- 172.21.15.139:0/3420036005 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6360005040 con 0x7f636c0f4aa0 2021-11-20T06:02:35.796 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.804+0000 7f636ad9d700 1 --2- 172.21.15.139:0/3420036005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636c0f4aa0 0x7f636c107150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f635c0041d0 tx=0x7f635c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:35.796 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.805+0000 7f63537fe700 1 -- 172.21.15.139:0/3420036005 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f635c0092e0 con 0x7f636c0f4aa0 2021-11-20T06:02:35.796 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.805+0000 7f6371053700 1 -- 172.21.15.139:0/3420036005 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636c1086d0 con 0x7f636c0f4aa0 2021-11-20T06:02:35.797 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.805+0000 7f63537fe700 1 -- 172.21.15.139:0/3420036005 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f635c00f040 con 0x7f636c0f4aa0 2021-11-20T06:02:35.797 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.805+0000 7f63537fe700 1 -- 172.21.15.139:0/3420036005 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f635c014850 con 0x7f636c0f4aa0 2021-11-20T06:02:35.797 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.805+0000 7f6371053700 1 -- 172.21.15.139:0/3420036005 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f636c108c10 con 0x7f636c0f4aa0 2021-11-20T06:02:35.798 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.806+0000 7f6371053700 1 -- 172.21.15.139:0/3420036005 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636c045ad0 con 0x7f636c0f4aa0 2021-11-20T06:02:35.799 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.806+0000 7f63537fe700 1 -- 172.21.15.139:0/3420036005 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f635c01b030 con 0x7f636c0f4aa0 2021-11-20T06:02:35.799 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.806+0000 7f63537fe700 1 --2- 172.21.15.139:0/3420036005 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f63540606b0 0x7f6354062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:35.799 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.806+0000 7f63537fe700 1 -- 172.21.15.139:0/3420036005 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f635c075b20 con 0x7f636c0f4aa0 2021-11-20T06:02:35.799 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.806+0000 7f636a59c700 1 --2- 172.21.15.139:0/3420036005 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f63540606b0 0x7f6354062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:35.800 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.807+0000 7f636a59c700 1 --2- 172.21.15.139:0/3420036005 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f63540606b0 0x7f6354062b70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f6360004cc0 tx=0x7f636000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:35.800 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.810+0000 7f63537fe700 1 -- 172.21.15.139:0/3420036005 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f635c048b20 con 0x7f636c0f4aa0 2021-11-20T06:02:35.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.962+0000 7f6371053700 1 -- 172.21.15.139:0/3420036005 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f636c0ff030 con 0x7f636c0f4aa0 2021-11-20T06:02:35.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.962+0000 7f63537fe700 1 -- 172.21.15.139:0/3420036005 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+10948 (secure 0 0 0) 0x7f635c019070 con 0x7f636c0f4aa0 2021-11-20T06:02:35.949 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:02:35.949 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":36,"fsid":"d819f2cc-49c6-11ec-8c2c-001a4aab830c","created":"2021-11-20T05:58:33.540072+0000","modified":"2021-11-20T06:02:28.340711+0000","last_up_change":"2021-11-20T06:02:27.182023+0000","last_in_change":"2021-11-20T06:02:11.372444+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T06:01:48.405544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"8fece5a6-7a49-451c-beb8-f90723d42033","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6803","nonce":22698595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6805","nonce":22698595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6809","nonce":22698595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6807","nonce":22698595}]},"public_addr":"172.21.15.139:6803/22698595","cluster_addr":"172.21.15.139:6805/22698595","heartbeat_back_addr":"172.21.15.139:6809/22698595","heartbeat_front_addr":"172.21.15.139:6807/22698595","state":["exists","up"]},{"osd":1,"uuid":"ce18ef72-7c6c-43f7-8a0c-6904f7c9e606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6811","nonce":2530906188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6813","nonce":2530906188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6817","nonce":2530906188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6815","nonce":2530906188}]},"public_addr":"172.21.15.139:6811/2530906188","cluster_addr":"172.21.15.139:6813/2530906188","heartbeat_back_addr":"172.21.15.139:6817/2530906188","heartbeat_front_addr":"172.21.15.139:6815/2530906188","state":["exists","up"]},{"osd":2,"uuid":"80cfe585-e283-45a4-9dd7-f328ec00d21c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6819","nonce":1147042903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6821","nonce":1147042903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6825","nonce":1147042903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6823","nonce":1147042903}]},"public_addr":"172.21.15.139:6819/1147042903","cluster_addr":"172.21.15.139:6821/1147042903","heartbeat_back_addr":"172.21.15.139:6825/1147042903","heartbeat_front_addr":"172.21.15.139:6823/1147042903","state":["exists","up"]},{"osd":3,"uuid":"793e383f-4d5a-41cb-b5b3-b4e4e07d6b8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6801","nonce":927121970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6803","nonce":927121970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6807","nonce":927121970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6805","nonce":927121970}]},"public_addr":"172.21.15.178:6801/927121970","cluster_addr":"172.21.15.178:6803/927121970","heartbeat_back_addr":"172.21.15.178:6807/927121970","heartbeat_front_addr":"172.21.15.178:6805/927121970","state":["exists","up"]},{"osd":4,"uuid":"3e453fbf-8ef9-4227-b3b8-cd6631bff6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6809","nonce":2601954583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6811","nonce":2601954583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6815","nonce":2601954583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6813","nonce":2601954583}]},"public_addr":"172.21.15.178:6809/2601954583","cluster_addr":"172.21.15.178:6811/2601954583","heartbeat_back_addr":"172.21.15.178:6815/2601954583","heartbeat_front_addr":"172.21.15.178:6813/2601954583","state":["exists","up"]},{"osd":5,"uuid":"b7d0eee4-3cb1-43dc-9c3f-f95593cf47cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6817","nonce":3932451665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6819","nonce":3932451665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6823","nonce":3932451665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6821","nonce":3932451665}]},"public_addr":"172.21.15.178:6817/3932451665","cluster_addr":"172.21.15.178:6819/3932451665","heartbeat_back_addr":"172.21.15.178:6823/3932451665","heartbeat_front_addr":"172.21.15.178:6821/3932451665","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:01.290566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:19.499151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:37.169640+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:52.838242+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:02:09.060491+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:02:25.766679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.139:0/3013495710":"2021-11-21T05:59:26.112534+0000","172.21.15.139:0/1461722096":"2021-11-21T05:59:13.525765+0000","172.21.15.139:0/1750602715":"2021-11-21T05:59:26.112534+0000","172.21.15.139:6800/3637001875":"2021-11-21T05:59:26.112534+0000","172.21.15.139:6801/1220659339":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6800/1220659339":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6801/3637001875":"2021-11-21T05:59:26.112534+0000","172.21.15.139:0/2668621147":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6800/4049861874":"2021-11-21T05:58:51.653389+0000","172.21.15.139:0/2863654682":"2021-11-21T05:58:51.653389+0000","172.21.15.139:0/2447016509":"2021-11-21T05:58:51.653389+0000","172.21.15.139:6801/4049861874":"2021-11-21T05:58:51.653389+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T06:02:35.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.964+0000 7f6371053700 1 -- 172.21.15.139:0/3420036005 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f63540606b0 msgr2=0x7f6354062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:35.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.964+0000 7f6371053700 1 --2- 172.21.15.139:0/3420036005 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f63540606b0 0x7f6354062b70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f6360004cc0 tx=0x7f636000b040).stop 2021-11-20T06:02:35.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.964+0000 7f6371053700 1 -- 172.21.15.139:0/3420036005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636c0f4aa0 msgr2=0x7f636c107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:35.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.964+0000 7f6371053700 1 --2- 172.21.15.139:0/3420036005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636c0f4aa0 0x7f636c107150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f635c0041d0 tx=0x7f635c00b040).stop 2021-11-20T06:02:35.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.964+0000 7f6371053700 1 -- 172.21.15.139:0/3420036005 shutdown_connections 2021-11-20T06:02:35.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.964+0000 7f6371053700 1 --2- 172.21.15.139:0/3420036005 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f63540606b0 0x7f6354062b70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:35.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.964+0000 7f6371053700 1 --2- 172.21.15.139:0/3420036005 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f636c0f4aa0 0x7f636c107150 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:35.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.964+0000 7f6371053700 1 --2- 172.21.15.139:0/3420036005 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f636c0f5440 0x7f636c107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:35.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.964+0000 7f6371053700 1 -- 172.21.15.139:0/3420036005 >> 172.21.15.139:0/3420036005 conn(0x7f636c0f05e0 msgr2=0x7f636c0fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:35.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.964+0000 7f6371053700 1 -- 172.21.15.139:0/3420036005 shutdown_connections 2021-11-20T06:02:35.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:35.964+0000 7f6371053700 1 -- 172.21.15.139:0/3420036005 wait complete. 2021-11-20T06:02:36.262 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:36 smithi139 conmon[33501]: cluster 2021-11-20T06:02:34.199594+0000 mgr.smithi139.dplyxn (mgr.14162) 150 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:36.262 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:36 smithi139 conmon[33501]: audit 2021-11-20T06:02:35.963107+0000 mon.smithi139 (mon.0) 435 : audit [DBG] from='client.? 172.21.15.139:0/3420036005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T06:02:36.273 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2021-11-20T06:02:36.274 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph osd dump --format=json 2021-11-20T06:02:36.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:36 smithi178 conmon[36600]: cluster 2021-11-20T06:02:34.199594+0000 mgr.smithi139.dplyxn (mgr.14162) 150 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:36.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:36 smithi178 conmon[36600]: audit 2021-11-20T06:02:35. 2021-11-20T06:02:36.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:36 smithi178 conmon[36600]: 963107+0000 mon.smithi139 (mon.0) 435 : audit [DBG] from='client.? 172.21.15.139:0/3420036005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T06:02:36.709 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:02:38.003 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.015+0000 7fdbc619a700 1 -- 172.21.15.139:0/1672230937 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbc00f6980 msgr2=0x7fdbc00f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:38.003 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.015+0000 7fdbc619a700 1 --2- 172.21.15.139:0/1672230937 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbc00f6980 0x7fdbc00f6de0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fdbb4004660 tx=0x7fdbb4009b30).stop 2021-11-20T06:02:38.004 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.015+0000 7fdbc619a700 1 -- 172.21.15.139:0/1672230937 shutdown_connections 2021-11-20T06:02:38.004 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.015+0000 7fdbc619a700 1 --2- 172.21.15.139:0/1672230937 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbc00f6980 0x7fdbc00f6de0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:38.004 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.015+0000 7fdbc619a700 1 --2- 172.21.15.139:0/1672230937 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc00f56e0 0x7fdbc00f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:38.005 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.015+0000 7fdbc619a700 1 -- 172.21.15.139:0/1672230937 >> 172.21.15.139:0/1672230937 conn(0x7fdbc00f0e20 msgr2=0x7fdbc00f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:38.005 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.015+0000 7fdbc619a700 1 -- 172.21.15.139:0/1672230937 shutdown_connections 2021-11-20T06:02:38.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.015+0000 7fdbc619a700 1 -- 172.21.15.139:0/1672230937 wait complete. 2021-11-20T06:02:38.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.016+0000 7fdbc619a700 1 Processor -- start 2021-11-20T06:02:38.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.016+0000 7fdbc619a700 1 -- start start 2021-11-20T06:02:38.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.016+0000 7fdbc619a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbc00f56e0 0x7fdbc0105fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:38.007 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.016+0000 7fdbc619a700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc00f6980 0x7fdbc0106490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:38.007 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.016+0000 7fdbc619a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc0106a40 con 0x7fdbc00f56e0 2021-11-20T06:02:38.007 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.016+0000 7fdbc619a700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc0106b80 con 0x7fdbc00f6980 2021-11-20T06:02:38.007 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.016+0000 7fdbbf7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbc00f56e0 0x7fdbc0105fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:38.008 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.016+0000 7fdbbf7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbc00f56e0 0x7fdbc0105fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57156/0 (socket says 172.21.15.139:57156) 2021-11-20T06:02:38.008 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.016+0000 7fdbbf7fe700 1 -- 172.21.15.139:0/3339592019 learned_addr learned my addr 172.21.15.139:0/3339592019 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:38.008 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.016+0000 7fdbbf7fe700 1 -- 172.21.15.139:0/3339592019 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc00f6980 msgr2=0x7fdbc0106490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:38.008 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.016+0000 7fdbbf7fe700 1 --2- 172.21.15.139:0/3339592019 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc00f6980 0x7fdbc0106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:02:38.009 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.016+0000 7fdbbf7fe700 1 -- 172.21.15.139:0/3339592019 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb4005040 con 0x7fdbc00f56e0 2021-11-20T06:02:38.009 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.017+0000 7fdbbf7fe700 1 --2- 172.21.15.139:0/3339592019 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbc00f56e0 0x7fdbc0105fa0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fdbb000c530 tx=0x7fdbb0007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:38.009 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.017+0000 7fdba7fff700 1 -- 172.21.15.139:0/3339592019 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbb000f650 con 0x7fdbc00f56e0 2021-11-20T06:02:38.009 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.017+0000 7fdba7fff700 1 -- 172.21.15.139:0/3339592019 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbb000fc60 con 0x7fdbc00f56e0 2021-11-20T06:02:38.010 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.017+0000 7fdbc619a700 1 -- 172.21.15.139:0/3339592019 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc0107520 con 0x7fdbc00f56e0 2021-11-20T06:02:38.010 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.017+0000 7fdba7fff700 1 -- 172.21.15.139:0/3339592019 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbb0012d20 con 0x7fdbc00f56e0 2021-11-20T06:02:38.010 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.017+0000 7fdbc619a700 1 -- 172.21.15.139:0/3339592019 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc0107ae0 con 0x7fdbc00f56e0 2021-11-20T06:02:38.011 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.018+0000 7fdbc619a700 1 -- 172.21.15.139:0/3339592019 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc0045ad0 con 0x7fdbc00f56e0 2021-11-20T06:02:38.012 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.018+0000 7fdba7fff700 1 -- 172.21.15.139:0/3339592019 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fdbb0019030 con 0x7fdbc00f56e0 2021-11-20T06:02:38.012 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.018+0000 7fdba7fff700 1 --2- 172.21.15.139:0/3339592019 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdba805c2a0 0x7fdba805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:38.012 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.019+0000 7fdba7fff700 1 -- 172.21.15.139:0/3339592019 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fdbb0075830 con 0x7fdbc00f56e0 2021-11-20T06:02:38.013 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.019+0000 7fdbbeffd700 1 --2- 172.21.15.139:0/3339592019 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdba805c2a0 0x7fdba805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:38.013 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.019+0000 7fdbbeffd700 1 --2- 172.21.15.139:0/3339592019 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdba805c2a0 0x7fdba805e760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fdbb4009f50 tx=0x7fdbb4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:38.013 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.022+0000 7fdba7fff700 1 -- 172.21.15.139:0/3339592019 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdbb0048880 con 0x7fdbc00f56e0 2021-11-20T06:02:38.159 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.174+0000 7fdbc619a700 1 -- 172.21.15.139:0/3339592019 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbc0027a00 con 0x7fdbc00f56e0 2021-11-20T06:02:38.160 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.174+0000 7fdba7fff700 1 -- 172.21.15.139:0/3339592019 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+10948 (secure 0 0 0) 0x7fdbc0027a00 con 0x7fdbc00f56e0 2021-11-20T06:02:38.160 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:02:38.161 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":36,"fsid":"d819f2cc-49c6-11ec-8c2c-001a4aab830c","created":"2021-11-20T05:58:33.540072+0000","modified":"2021-11-20T06:02:28.340711+0000","last_up_change":"2021-11-20T06:02:27.182023+0000","last_in_change":"2021-11-20T06:02:11.372444+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T06:01:48.405544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"8fece5a6-7a49-451c-beb8-f90723d42033","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6803","nonce":22698595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6805","nonce":22698595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6809","nonce":22698595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6807","nonce":22698595}]},"public_addr":"172.21.15.139:6803/22698595","cluster_addr":"172.21.15.139:6805/22698595","heartbeat_back_addr":"172.21.15.139:6809/22698595","heartbeat_front_addr":"172.21.15.139:6807/22698595","state":["exists","up"]},{"osd":1,"uuid":"ce18ef72-7c6c-43f7-8a0c-6904f7c9e606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6811","nonce":2530906188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6813","nonce":2530906188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6817","nonce":2530906188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6815","nonce":2530906188}]},"public_addr":"172.21.15.139:6811/2530906188","cluster_addr":"172.21.15.139:6813/2530906188","heartbeat_back_addr":"172.21.15.139:6817/2530906188","heartbeat_front_addr":"172.21.15.139:6815/2530906188","state":["exists","up"]},{"osd":2,"uuid":"80cfe585-e283-45a4-9dd7-f328ec00d21c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6819","nonce":1147042903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6821","nonce":1147042903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6825","nonce":1147042903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6823","nonce":1147042903}]},"public_addr":"172.21.15.139:6819/1147042903","cluster_addr":"172.21.15.139:6821/1147042903","heartbeat_back_addr":"172.21.15.139:6825/1147042903","heartbeat_front_addr":"172.21.15.139:6823/1147042903","state":["exists","up"]},{"osd":3,"uuid":"793e383f-4d5a-41cb-b5b3-b4e4e07d6b8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6801","nonce":927121970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6803","nonce":927121970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6807","nonce":927121970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6805","nonce":927121970}]},"public_addr":"172.21.15.178:6801/927121970","cluster_addr":"172.21.15.178:6803/927121970","heartbeat_back_addr":"172.21.15.178:6807/927121970","heartbeat_front_addr":"172.21.15.178:6805/927121970","state":["exists","up"]},{"osd":4,"uuid":"3e453fbf-8ef9-4227-b3b8-cd6631bff6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6809","nonce":2601954583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6811","nonce":2601954583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6815","nonce":2601954583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6813","nonce":2601954583}]},"public_addr":"172.21.15.178:6809/2601954583","cluster_addr":"172.21.15.178:6811/2601954583","heartbeat_back_addr":"172.21.15.178:6815/2601954583","heartbeat_front_addr":"172.21.15.178:6813/2601954583","state":["exists","up"]},{"osd":5,"uuid":"b7d0eee4-3cb1-43dc-9c3f-f95593cf47cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6817","nonce":3932451665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6819","nonce":3932451665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6823","nonce":3932451665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6821","nonce":3932451665}]},"public_addr":"172.21.15.178:6817/3932451665","cluster_addr":"172.21.15.178:6819/3932451665","heartbeat_back_addr":"172.21.15.178:6823/3932451665","heartbeat_front_addr":"172.21.15.178:6821/3932451665","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:01.290566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:19.499151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:37.169640+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:52.838242+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:02:09.060491+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:02:25.766679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.139:0/3013495710":"2021-11-21T05:59:26.112534+0000","172.21.15.139:0/1461722096":"2021-11-21T05:59:13.525765+0000","172.21.15.139:0/1750602715":"2021-11-21T05:59:26.112534+0000","172.21.15.139:6800/3637001875":"2021-11-21T05:59:26.112534+0000","172.21.15.139:6801/1220659339":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6800/1220659339":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6801/3637001875":"2021-11-21T05:59:26.112534+0000","172.21.15.139:0/2668621147":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6800/4049861874":"2021-11-21T05:58:51.653389+0000","172.21.15.139:0/2863654682":"2021-11-21T05:58:51.653389+0000","172.21.15.139:0/2447016509":"2021-11-21T05:58:51.653389+0000","172.21.15.139:6801/4049861874":"2021-11-21T05:58:51.653389+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T06:02:38.162 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.176+0000 7fdbc619a700 1 -- 172.21.15.139:0/3339592019 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdba805c2a0 msgr2=0x7fdba805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:38.162 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.176+0000 7fdbc619a700 1 --2- 172.21.15.139:0/3339592019 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdba805c2a0 0x7fdba805e760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fdbb4009f50 tx=0x7fdbb4009b30).stop 2021-11-20T06:02:38.162 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.176+0000 7fdbc619a700 1 -- 172.21.15.139:0/3339592019 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbc00f56e0 msgr2=0x7fdbc0105fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:38.162 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.176+0000 7fdbc619a700 1 --2- 172.21.15.139:0/3339592019 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbc00f56e0 0x7fdbc0105fa0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fdbb000c530 tx=0x7fdbb0007e10).stop 2021-11-20T06:02:38.163 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.176+0000 7fdbc619a700 1 -- 172.21.15.139:0/3339592019 shutdown_connections 2021-11-20T06:02:38.163 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.176+0000 7fdbc619a700 1 --2- 172.21.15.139:0/3339592019 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdba805c2a0 0x7fdba805e760 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:38.163 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.176+0000 7fdbc619a700 1 --2- 172.21.15.139:0/3339592019 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdbc00f56e0 0x7fdbc0105fa0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:38.164 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.176+0000 7fdbc619a700 1 --2- 172.21.15.139:0/3339592019 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc00f6980 0x7fdbc0106490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:02:38.164 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.176+0000 7fdbc619a700 1 -- 172.21.15.139:0/3339592019 >> 172.21.15.139:0/3339592019 conn(0x7fdbc00f0e20 msgr2=0x7fdbc00fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:38.164 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.176+0000 7fdbc619a700 1 -- 172.21.15.139:0/3339592019 shutdown_connections 2021-11-20T06:02:38.164 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:38.176+0000 7fdbc619a700 1 -- 172.21.15.139:0/3339592019 wait complete. 2021-11-20T06:02:38.277 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:38 smithi139 conmon[33501]: cluster 2021-11-20T06:02:36.199908+0000 mgr.smithi139.dplyxn (mgr.14162) 151 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:38.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:38 smithi178 conmon[36600]: cluster 2021-11-20T06:02:36.199908+0000 mgr.smithi139.dplyxn (mgr.14162) 151 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:38.507 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2021-11-20T06:02:38.942 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:02:39.337 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:39 smithi139 conmon[33501]: audit 2021-11-20T06:02:38.174949+0000 mon.smithi139 (mon 2021-11-20T06:02:39.338 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:39 smithi139 conmon[33501]: .0) 436 : audit [DBG] from='client.? 172.21.15.139:0/3339592019' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T06:02:39.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:39 smithi178 conmon[36600]: audit 2021-11-20T06:02:38.174949+0000 mon.smithi139 (mon.0) 436 : audit [DBG] from='client.? 172.21.15.139:0/3339592019' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T06:02:40.083 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.097+0000 7f71ff3a1700 1 -- 172.21.15.139:0/1291051482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71f80f62f0 msgr2=0x7f71f80f6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:40.083 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.097+0000 7f71ff3a1700 1 --2- 172.21.15.139:0/1291051482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71f80f62f0 0x7f71f80f6750 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f71e8004660 tx=0x7f71e8009b30).stop 2021-11-20T06:02:40.084 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.097+0000 7f71ff3a1700 1 -- 172.21.15.139:0/1291051482 shutdown_connections 2021-11-20T06:02:40.084 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.097+0000 7f71ff3a1700 1 --2- 172.21.15.139:0/1291051482 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71f80f62f0 0x7f71f80f6750 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:40.084 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.097+0000 7f71ff3a1700 1 --2- 172.21.15.139:0/1291051482 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71f80f7ce0 0x7f71f80f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:02:40.084 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.097+0000 7f71ff3a1700 1 -- 172.21.15.139:0/1291051482 >> 172.21.15.139:0/1291051482 conn(0x7f71f80f1d70 msgr2=0x7f71f80f41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:40.085 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.098+0000 7f71ff3a1700 1 -- 172.21.15.139:0/1291051482 shutdown_connections 2021-11-20T06:02:40.085 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.098+0000 7f71ff3a1700 1 -- 172.21.15.139:0/1291051482 wait complete. 2021-11-20T06:02:40.088 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.098+0000 7f71ff3a1700 1 Processor -- start 2021-11-20T06:02:40.088 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.098+0000 7f71ff3a1700 1 -- start start 2021-11-20T06:02:40.089 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.098+0000 7f71ff3a1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71f80f62f0 0x7f71f8107000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:40.089 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.098+0000 7f71ff3a1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71f80f7ce0 0x7f71f81074f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:40.089 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.098+0000 7f71ff3a1700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f8107b30 con 0x7f71f80f7ce0 2021-11-20T06:02:40.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.098+0000 7f71ff3a1700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f81083e0 con 0x7f71f80f62f0 2021-11-20T06:02:40.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.099+0000 7f71fdb9e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71f80f7ce0 0x7f71f81074f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:40.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.099+0000 7f71fdb9e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71f80f7ce0 0x7f71f81074f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57168/0 (socket says 172.21.15.139:57168) 2021-11-20T06:02:40.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.099+0000 7f71fdb9e700 1 -- 172.21.15.139:0/3206951596 learned_addr learned my addr 172.21.15.139:0/3206951596 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:40.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.099+0000 7f71fe39f700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71f80f62f0 0x7f71f8107000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:40.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.099+0000 7f71fdb9e700 1 -- 172.21.15.139:0/3206951596 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71f80f62f0 msgr2=0x7f71f8107000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:40.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.099+0000 7f71fdb9e700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71f80f62f0 0x7f71f8107000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:40.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.099+0000 7f71fdb9e700 1 -- 172.21.15.139:0/3206951596 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e8005040 con 0x7f71f80f7ce0 2021-11-20T06:02:40.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.099+0000 7f71fe39f700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71f80f62f0 0x7f71f8107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:02:40.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.099+0000 7f71fdb9e700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71f80f7ce0 0x7f71f81074f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f71e8009830 tx=0x7f71e8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:40.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.099+0000 7f71eeffd700 1 -- 172.21.15.139:0/3206951596 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71e8018070 con 0x7f71f80f7ce0 2021-11-20T06:02:40.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.099+0000 7f71eeffd700 1 -- 172.21.15.139:0/3206951596 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71e8013410 con 0x7f71f80f7ce0 2021-11-20T06:02:40.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.099+0000 7f71ff3a1700 1 -- 172.21.15.139:0/3206951596 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f8108660 con 0x7f71f80f7ce0 2021-11-20T06:02:40.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.099+0000 7f71eeffd700 1 -- 172.21.15.139:0/3206951596 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71e801c620 con 0x7f71f80f7ce0 2021-11-20T06:02:40.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.099+0000 7f71ff3a1700 1 -- 172.21.15.139:0/3206951596 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f8108bc0 con 0x7f71f80f7ce0 2021-11-20T06:02:40.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.100+0000 7f71ff3a1700 1 -- 172.21.15.139:0/3206951596 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f71f8059100 con 0x7f71f80f7ce0 2021-11-20T06:02:40.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.101+0000 7f71eeffd700 1 -- 172.21.15.139:0/3206951596 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f71e801c890 con 0x7f71f80f7ce0 2021-11-20T06:02:40.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.101+0000 7f71eeffd700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f71e405c2b0 0x7f71e405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:40.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.101+0000 7f71eeffd700 1 -- 172.21.15.139:0/3206951596 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f71e8077eb0 con 0x7f71f80f7ce0 2021-11-20T06:02:40.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.101+0000 7f71fe39f700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f71e405c2b0 0x7f71e405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:40.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.101+0000 7f71eeffd700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595] conn(0x7f71e4062040 0x7f71e40644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:40.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.101+0000 7f71eeffd700 1 -- 172.21.15.139:0/3206951596 --> [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71e4064b20 con 0x7f71e4062040 2021-11-20T06:02:40.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.101+0000 7f71eeffd700 1 -- 172.21.15.139:0/3206951596 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71e80782b0 con 0x7f71f80f7ce0 2021-11-20T06:02:40.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.101+0000 7f71feba0700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595] conn(0x7f71e4062040 0x7f71e40644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:40.096 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.101+0000 7f71fe39f700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f71e405c2b0 0x7f71e405e770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f71f80f7730 tx=0x7f71f4008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:40.096 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.102+0000 7f71feba0700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595] conn(0x7f71e4062040 0x7f71e40644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:40.096 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.102+0000 7f71eeffd700 1 -- 172.21.15.139:0/3206951596 <== osd.0 v2:172.21.15.139:6802/22698595 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f71e4064b20 con 0x7f71e4062040 2021-11-20T06:02:40.102 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.116+0000 7f71ff3a1700 1 -- 172.21.15.139:0/3206951596 --> [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f71f8109130 con 0x7f71e4062040 2021-11-20T06:02:40.102 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.117+0000 7f71eeffd700 1 -- 172.21.15.139:0/3206951596 <== osd.0 v2:172.21.15.139:6802/22698595 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f71f8109130 con 0x7f71e4062040 2021-11-20T06:02:40.104 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.117+0000 7f71ff3a1700 1 -- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595] conn(0x7f71e4062040 msgr2=0x7f71e40644a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:40.104 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.117+0000 7f71ff3a1700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595] conn(0x7f71e4062040 0x7f71e40644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:40.104 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.117+0000 7f71ff3a1700 1 -- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f71e405c2b0 msgr2=0x7f71e405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:40.104 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.117+0000 7f71ff3a1700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f71e405c2b0 0x7f71e405e770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f71f80f7730 tx=0x7f71f4008040).stop 2021-11-20T06:02:40.105 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.117+0000 7f71ff3a1700 1 -- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71f80f7ce0 msgr2=0x7f71f81074f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:40.105 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.117+0000 7f71ff3a1700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71f80f7ce0 0x7f71f81074f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f71e8009830 tx=0x7f71e8009b30).stop 2021-11-20T06:02:40.105 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.117+0000 7f71ff3a1700 1 -- 172.21.15.139:0/3206951596 shutdown_connections 2021-11-20T06:02:40.105 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.117+0000 7f71ff3a1700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:6802/22698595,v1:172.21.15.139:6803/22698595] conn(0x7f71e4062040 0x7f71e40644a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:40.105 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.117+0000 7f71ff3a1700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f71e405c2b0 0x7f71e405e770 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:40.106 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.117+0000 7f71ff3a1700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71f80f62f0 0x7f71f8107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:40.106 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.117+0000 7f71ff3a1700 1 --2- 172.21.15.139:0/3206951596 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f71f80f7ce0 0x7f71f81074f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:40.106 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.117+0000 7f71ff3a1700 1 -- 172.21.15.139:0/3206951596 >> 172.21.15.139:0/3206951596 conn(0x7f71f80f1d70 msgr2=0x7f71f80fb320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:40.106 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.117+0000 7f71ff3a1700 1 -- 172.21.15.139:0/3206951596 shutdown_connections 2021-11-20T06:02:40.107 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:40.117+0000 7f71ff3a1700 1 -- 172.21.15.139:0/3206951596 wait complete. 2021-11-20T06:02:40.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:40 smithi178 conmon[36600]: cluster 2021-11-20T06:02:38.200202+0000 mgr.smithi139.dplyxn (mgr.14162) 152 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:40.450 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:40 smithi139 conmon[33501]: cluster 2021-11-20T06:02:38.200202+0000 mgr.smithi139.dplyxn (mgr.14162) 152 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:40.451 INFO:teuthology.orchestra.run.smithi139.stdout:38654705686 2021-11-20T06:02:40.452 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2021-11-20T06:02:40.890 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:02:41.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:41 smithi139 conmon[33501]: cluster 2021-11-20T06:02:40.200526+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:02:41.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:41 smithi139 conmon[33501]: 14162) 153 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:41.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:41 smithi178 conmon[36600]: cluster 2021-11-20T06:02:40.200526+0000 mgr.smithi139.dplyxn (mgr.14162) 153 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:42.317 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.329+0000 7f6ea4d56700 1 -- 172.21.15.139:0/3201176884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ea00f5280 msgr2=0x7f6ea00f7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:42.318 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.329+0000 7f6ea4d56700 1 --2- 172.21.15.139:0/3201176884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ea00f5280 0x7f6ea00f7a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f6e94004660 tx=0x7f6e94009b30).stop 2021-11-20T06:02:42.318 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.329+0000 7f6ea4d56700 1 -- 172.21.15.139:0/3201176884 shutdown_connections 2021-11-20T06:02:42.318 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.329+0000 7f6ea4d56700 1 --2- 172.21.15.139:0/3201176884 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ea00f5280 0x7f6ea00f7a00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:42.319 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.329+0000 7f6ea4d56700 1 --2- 172.21.15.139:0/3201176884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ea00f48e0 0x7f6ea00f4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:42.319 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.329+0000 7f6ea4d56700 1 -- 172.21.15.139:0/3201176884 >> 172.21.15.139:0/3201176884 conn(0x7f6ea00f0400 msgr2=0x7f6ea00f2820 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:42.319 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.330+0000 7f6ea4d56700 1 -- 172.21.15.139:0/3201176884 shutdown_connections 2021-11-20T06:02:42.319 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.330+0000 7f6ea4d56700 1 -- 172.21.15.139:0/3201176884 wait complete. 2021-11-20T06:02:42.319 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.330+0000 7f6ea4d56700 1 Processor -- start 2021-11-20T06:02:42.320 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.330+0000 7f6ea4d56700 1 -- start start 2021-11-20T06:02:42.320 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.330+0000 7f6ea4d56700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ea00f48e0 0x7f6ea00f7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:42.320 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.330+0000 7f6ea4d56700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ea00f5280 0x7f6ea00f5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:42.320 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.330+0000 7f6ea4d56700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea010abc0 con 0x7f6ea00f5280 2021-11-20T06:02:42.321 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.330+0000 7f6ea4d56700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea010ad00 con 0x7f6ea00f48e0 2021-11-20T06:02:42.321 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.331+0000 7f6e9effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ea00f5280 0x7f6ea00f5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:42.321 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.331+0000 7f6e9effd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ea00f5280 0x7f6ea00f5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57180/0 (socket says 172.21.15.139:57180) 2021-11-20T06:02:42.321 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.331+0000 7f6e9effd700 1 -- 172.21.15.139:0/206022031 learned_addr learned my addr 172.21.15.139:0/206022031 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:42.321 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.331+0000 7f6e9f7fe700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ea00f48e0 0x7f6ea00f7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:42.322 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.331+0000 7f6e9effd700 1 -- 172.21.15.139:0/206022031 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ea00f48e0 msgr2=0x7f6ea00f7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:42.322 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.331+0000 7f6e9effd700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ea00f48e0 0x7f6ea00f7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:42.322 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.331+0000 7f6e9effd700 1 -- 172.21.15.139:0/206022031 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e94005040 con 0x7f6ea00f5280 2021-11-20T06:02:42.322 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.331+0000 7f6e9effd700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ea00f5280 0x7f6ea00f5910 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f6e94009f50 tx=0x7f6e94009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:42.323 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.331+0000 7f6e87fff700 1 -- 172.21.15.139:0/206022031 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e94018070 con 0x7f6ea00f5280 2021-11-20T06:02:42.323 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.331+0000 7f6e87fff700 1 -- 172.21.15.139:0/206022031 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e94013410 con 0x7f6ea00f5280 2021-11-20T06:02:42.323 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.331+0000 7f6ea4d56700 1 -- 172.21.15.139:0/206022031 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea00f5e60 con 0x7f6ea00f5280 2021-11-20T06:02:42.323 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.331+0000 7f6e87fff700 1 -- 172.21.15.139:0/206022031 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e9401c3c0 con 0x7f6ea00f5280 2021-11-20T06:02:42.324 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.331+0000 7f6ea4d56700 1 -- 172.21.15.139:0/206022031 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea00f63c0 con 0x7f6ea00f5280 2021-11-20T06:02:42.324 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.332+0000 7f6ea4d56700 1 -- 172.21.15.139:0/206022031 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ea005db60 con 0x7f6ea00f5280 2021-11-20T06:02:42.325 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.333+0000 7f6e87fff700 1 -- 172.21.15.139:0/206022031 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6e9401d420 con 0x7f6ea00f5280 2021-11-20T06:02:42.325 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.333+0000 7f6e87fff700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6e8805c2b0 0x7f6e8805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:42.326 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.333+0000 7f6e87fff700 1 -- 172.21.15.139:0/206022031 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f6e940781a0 con 0x7f6ea00f5280 2021-11-20T06:02:42.326 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.333+0000 7f6e9f7fe700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6e8805c2b0 0x7f6e8805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:42.326 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.333+0000 7f6e87fff700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:6810/2530906188,v1:172.21.15.139:6811/2530906188] conn(0x7f6e88062040 0x7f6e880644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:42.326 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.333+0000 7f6e87fff700 1 -- 172.21.15.139:0/206022031 --> [v2:172.21.15.139:6810/2530906188,v1:172.21.15.139:6811/2530906188] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e88064b20 con 0x7f6e88062040 2021-11-20T06:02:42.327 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.333+0000 7f6e87fff700 1 -- 172.21.15.139:0/206022031 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e940785a0 con 0x7f6ea00f5280 2021-11-20T06:02:42.327 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.333+0000 7f6e9ffff700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:6810/2530906188,v1:172.21.15.139:6811/2530906188] conn(0x7f6e88062040 0x7f6e880644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:42.327 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.334+0000 7f6e9f7fe700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6e8805c2b0 0x7f6e8805e770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f6ea00f7220 tx=0x7f6e90009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:42.327 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.334+0000 7f6e9ffff700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:6810/2530906188,v1:172.21.15.139:6811/2530906188] conn(0x7f6e88062040 0x7f6e880644a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:42.327 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.334+0000 7f6e87fff700 1 -- 172.21.15.139:0/206022031 <== osd.1 v2:172.21.15.139:6810/2530906188 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f6e88064b20 con 0x7f6e88062040 2021-11-20T06:02:42.334 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.348+0000 7f6ea4d56700 1 -- 172.21.15.139:0/206022031 --> [v2:172.21.15.139:6810/2530906188,v1:172.21.15.139:6811/2530906188] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6ea0059100 con 0x7f6e88062040 2021-11-20T06:02:42.334 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.348+0000 7f6e87fff700 1 -- 172.21.15.139:0/206022031 <== osd.1 v2:172.21.15.139:6810/2530906188 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6ea0059100 con 0x7f6e88062040 2021-11-20T06:02:42.335 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.349+0000 7f6ea4d56700 1 -- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:6810/2530906188,v1:172.21.15.139:6811/2530906188] conn(0x7f6e88062040 msgr2=0x7f6e880644a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:42.335 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.349+0000 7f6ea4d56700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:6810/2530906188,v1:172.21.15.139:6811/2530906188] conn(0x7f6e88062040 0x7f6e880644a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:42.335 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.349+0000 7f6ea4d56700 1 -- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6e8805c2b0 msgr2=0x7f6e8805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:42.336 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.349+0000 7f6ea4d56700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6e8805c2b0 0x7f6e8805e770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f6ea00f7220 tx=0x7f6e90009040).stop 2021-11-20T06:02:42.336 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.349+0000 7f6ea4d56700 1 -- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ea00f5280 msgr2=0x7f6ea00f5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:42.336 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.349+0000 7f6ea4d56700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ea00f5280 0x7f6ea00f5910 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f6e94009f50 tx=0x7f6e94009b30).stop 2021-11-20T06:02:42.336 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.349+0000 7f6ea4d56700 1 -- 172.21.15.139:0/206022031 shutdown_connections 2021-11-20T06:02:42.337 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.349+0000 7f6ea4d56700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:6810/2530906188,v1:172.21.15.139:6811/2530906188] conn(0x7f6e88062040 0x7f6e880644a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:42.337 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.349+0000 7f6ea4d56700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6e8805c2b0 0x7f6e8805e770 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:42.337 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.349+0000 7f6ea4d56700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ea00f48e0 0x7f6ea00f7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:42.337 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.349+0000 7f6ea4d56700 1 --2- 172.21.15.139:0/206022031 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6ea00f5280 0x7f6ea00f5910 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:42.337 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.349+0000 7f6ea4d56700 1 -- 172.21.15.139:0/206022031 >> 172.21.15.139:0/206022031 conn(0x7f6ea00f0400 msgr2=0x7f6ea00fb450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:42.338 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.349+0000 7f6ea4d56700 1 -- 172.21.15.139:0/206022031 shutdown_connections 2021-11-20T06:02:42.338 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:42.349+0000 7f6ea4d56700 1 -- 172.21.15.139:0/206022031 wait complete. 2021-11-20T06:02:42.690 INFO:teuthology.orchestra.run.smithi139.stdout:60129542164 2021-11-20T06:02:42.690 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2021-11-20T06:02:43.124 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:02:44.275 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.287+0000 7f98698d9700 1 -- 172.21.15.139:0/2006671113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98640f56e0 msgr2=0x7f98640f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:44.275 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.287+0000 7f98698d9700 1 --2- 172.21.15.139:0/2006671113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98640f56e0 0x7f98640f5b00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f9854004660 tx=0x7f9854009b30).stop 2021-11-20T06:02:44.275 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.287+0000 7f98698d9700 1 -- 172.21.15.139:0/2006671113 shutdown_connections 2021-11-20T06:02:44.276 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.287+0000 7f98698d9700 1 --2- 172.21.15.139:0/2006671113 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98640f6980 0x7f98640f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:44.276 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.287+0000 7f98698d9700 1 --2- 172.21.15.139:0/2006671113 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98640f56e0 0x7f98640f5b00 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:44.276 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.287+0000 7f98698d9700 1 -- 172.21.15.139:0/2006671113 >> 172.21.15.139:0/2006671113 conn(0x7f98640f0e20 msgr2=0x7f98640f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:44.276 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.287+0000 7f98698d9700 1 -- 172.21.15.139:0/2006671113 shutdown_connections 2021-11-20T06:02:44.276 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.287+0000 7f98698d9700 1 -- 172.21.15.139:0/2006671113 wait complete. 2021-11-20T06:02:44.277 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.288+0000 7f98698d9700 1 Processor -- start 2021-11-20T06:02:44.277 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.288+0000 7f98698d9700 1 -- start start 2021-11-20T06:02:44.277 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.288+0000 7f98698d9700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98640f56e0 0x7f9864106000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:44.277 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.288+0000 7f98698d9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98640f6980 0x7f98641064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:44.277 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.288+0000 7f98698d9700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9864106aa0 con 0x7f98640f6980 2021-11-20T06:02:44.278 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.288+0000 7f98698d9700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9864106be0 con 0x7f98640f56e0 2021-11-20T06:02:44.278 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.288+0000 7f98627fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98640f6980 0x7f98641064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:44.278 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.288+0000 7f98627fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98640f6980 0x7f98641064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57190/0 (socket says 172.21.15.139:57190) 2021-11-20T06:02:44.278 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.288+0000 7f98627fc700 1 -- 172.21.15.139:0/2886645377 learned_addr learned my addr 172.21.15.139:0/2886645377 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:44.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.288+0000 7f9862ffd700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98640f56e0 0x7f9864106000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:44.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.288+0000 7f98627fc700 1 -- 172.21.15.139:0/2886645377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98640f56e0 msgr2=0x7f9864106000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:44.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.288+0000 7f98627fc700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98640f56e0 0x7f9864106000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:44.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.288+0000 7f98627fc700 1 -- 172.21.15.139:0/2886645377 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9854005040 con 0x7f98640f6980 2021-11-20T06:02:44.280 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.289+0000 7f98627fc700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98640f6980 0x7f98641064f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f985800c790 tx=0x7f9858007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:44.280 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.289+0000 7f984b7fe700 1 -- 172.21.15.139:0/2886645377 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f985800e680 con 0x7f98640f6980 2021-11-20T06:02:44.280 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.289+0000 7f98698d9700 1 -- 172.21.15.139:0/2886645377 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9864107580 con 0x7f98640f6980 2021-11-20T06:02:44.280 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.289+0000 7f984b7fe700 1 -- 172.21.15.139:0/2886645377 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f985800ec90 con 0x7f98640f6980 2021-11-20T06:02:44.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.289+0000 7f984b7fe700 1 -- 172.21.15.139:0/2886645377 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9858013d60 con 0x7f98640f6980 2021-11-20T06:02:44.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.289+0000 7f98698d9700 1 -- 172.21.15.139:0/2886645377 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9864107b40 con 0x7f98640f6980 2021-11-20T06:02:44.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.290+0000 7f98698d9700 1 -- 172.21.15.139:0/2886645377 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f986405dcf0 con 0x7f98640f6980 2021-11-20T06:02:44.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.290+0000 7f984b7fe700 1 -- 172.21.15.139:0/2886645377 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f985801a060 con 0x7f98640f6980 2021-11-20T06:02:44.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.290+0000 7f984b7fe700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f984c05c2a0 0x7f984c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:44.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.290+0000 7f984b7fe700 1 -- 172.21.15.139:0/2886645377 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f9858075de0 con 0x7f98640f6980 2021-11-20T06:02:44.284 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.290+0000 7f9862ffd700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f984c05c2a0 0x7f984c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:44.284 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.291+0000 7f984b7fe700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903] conn(0x7f984c062030 0x7f984c064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:44.284 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.291+0000 7f984b7fe700 1 -- 172.21.15.139:0/2886645377 --> [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f984c064b10 con 0x7f984c062030 2021-11-20T06:02:44.284 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.291+0000 7f984b7fe700 1 -- 172.21.15.139:0/2886645377 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9858078450 con 0x7f98640f6980 2021-11-20T06:02:44.284 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.291+0000 7f98637fe700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903] conn(0x7f984c062030 0x7f984c064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:44.285 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.291+0000 7f9862ffd700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f984c05c2a0 0x7f984c05e760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f9854009d60 tx=0x7f9854009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:44.285 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.291+0000 7f98637fe700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903] conn(0x7f984c062030 0x7f984c064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:44.285 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.292+0000 7f984b7fe700 1 -- 172.21.15.139:0/2886645377 <== osd.2 v2:172.21.15.139:6818/1147042903 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f984c064b10 con 0x7f984c062030 2021-11-20T06:02:44.291 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.305+0000 7f98698d9700 1 -- 172.21.15.139:0/2886645377 --> [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9864107fa0 con 0x7f984c062030 2021-11-20T06:02:44.292 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.306+0000 7f984b7fe700 1 -- 172.21.15.139:0/2886645377 <== osd.2 v2:172.21.15.139:6818/1147042903 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9864107fa0 con 0x7f984c062030 2021-11-20T06:02:44.293 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.306+0000 7f98698d9700 1 -- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903] conn(0x7f984c062030 msgr2=0x7f984c064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:44.293 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.306+0000 7f98698d9700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903] conn(0x7f984c062030 0x7f984c064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:44.294 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.306+0000 7f98698d9700 1 -- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f984c05c2a0 msgr2=0x7f984c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:44.294 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.306+0000 7f98698d9700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f984c05c2a0 0x7f984c05e760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f9854009d60 tx=0x7f9854009b30).stop 2021-11-20T06:02:44.295 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.306+0000 7f98698d9700 1 -- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98640f6980 msgr2=0x7f98641064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:44.295 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.306+0000 7f98698d9700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98640f6980 0x7f98641064f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f985800c790 tx=0x7f9858007d70).stop 2021-11-20T06:02:44.296 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.306+0000 7f98698d9700 1 -- 172.21.15.139:0/2886645377 shutdown_connections 2021-11-20T06:02:44.296 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.306+0000 7f98698d9700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:6818/1147042903,v1:172.21.15.139:6819/1147042903] conn(0x7f984c062030 0x7f984c064490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:44.296 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.306+0000 7f98698d9700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f984c05c2a0 0x7f984c05e760 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:44.297 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.306+0000 7f98698d9700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98640f56e0 0x7f9864106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:44.297 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.306+0000 7f98698d9700 1 --2- 172.21.15.139:0/2886645377 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98640f6980 0x7f98641064f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:44.297 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.306+0000 7f98698d9700 1 -- 172.21.15.139:0/2886645377 >> 172.21.15.139:0/2886645377 conn(0x7f98640f0e20 msgr2=0x7f98640fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:44.298 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.306+0000 7f98698d9700 1 -- 172.21.15.139:0/2886645377 shutdown_connections 2021-11-20T06:02:44.298 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:44.306+0000 7f98698d9700 1 -- 172.21.15.139:0/2886645377 wait complete. 2021-11-20T06:02:44.423 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:44 smithi139 conmon[33501]: cluster 2021-11-20T06:02:42.200838+0000 mgr.smithi139.dplyxn (mgr.14162) 154 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:44.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:44 smithi178 conmon[36600]: cluster 2021-11-20T06:02:42.200838+0000 mgr.smithi139.dplyxn (mgr.14162) 154 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:44.621 INFO:teuthology.orchestra.run.smithi139.stdout:77309411343 2021-11-20T06:02:44.621 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2021-11-20T06:02:45.057 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:02:46.229 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.237+0000 7f73135f1700 1 -- 172.21.15.139:0/739406374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f730c0f5500 msgr2=0x7f730c0fbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:46.230 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.237+0000 7f73135f1700 1 --2- 172.21.15.139:0/739406374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f730c0f5500 0x7f730c0fbe10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f7300004660 tx=0x7f7300009b30).stop 2021-11-20T06:02:46.230 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.237+0000 7f73135f1700 1 -- 172.21.15.139:0/739406374 shutdown_connections 2021-11-20T06:02:46.230 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.237+0000 7f73135f1700 1 --2- 172.21.15.139:0/739406374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f730c0f5500 0x7f730c0fbe10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:46.231 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.237+0000 7f73135f1700 1 --2- 172.21.15.139:0/739406374 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f730c0f4b60 0x7f730c0f4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:46.231 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.237+0000 7f73135f1700 1 -- 172.21.15.139:0/739406374 >> 172.21.15.139:0/739406374 conn(0x7f730c0f05e0 msgr2=0x7f730c0f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:46.231 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.237+0000 7f73135f1700 1 -- 172.21.15.139:0/739406374 shutdown_connections 2021-11-20T06:02:46.231 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.237+0000 7f73135f1700 1 -- 172.21.15.139:0/739406374 wait complete. 2021-11-20T06:02:46.232 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.238+0000 7f73135f1700 1 Processor -- start 2021-11-20T06:02:46.233 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.238+0000 7f73135f1700 1 -- start start 2021-11-20T06:02:46.233 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.238+0000 7f73135f1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f730c0f4b60 0x7f730c0f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:46.233 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.238+0000 7f73135f1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f730c0f5500 0x7f730c0f5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:46.233 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.238+0000 7f73135f1700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c0f6130 con 0x7f730c0f5500 2021-11-20T06:02:46.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.238+0000 7f73135f1700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c0f6270 con 0x7f730c0f4b60 2021-11-20T06:02:46.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.238+0000 7f7310b8c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f730c0f5500 0x7f730c0f5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:46.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.238+0000 7f7310b8c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f730c0f5500 0x7f730c0f5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57202/0 (socket says 172.21.15.139:57202) 2021-11-20T06:02:46.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.238+0000 7f7310b8c700 1 -- 172.21.15.139:0/1891625427 learned_addr learned my addr 172.21.15.139:0/1891625427 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:46.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.238+0000 7f7310b8c700 1 -- 172.21.15.139:0/1891625427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f730c0f4b60 msgr2=0x7f730c0f7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:46.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.238+0000 7f731138d700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f730c0f4b60 0x7f730c0f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:46.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.238+0000 7f7310b8c700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f730c0f4b60 0x7f730c0f7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:46.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.238+0000 7f7310b8c700 1 -- 172.21.15.139:0/1891625427 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7300005040 con 0x7f730c0f5500 2021-11-20T06:02:46.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.239+0000 7f7310b8c700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f730c0f5500 0x7f730c0f5bd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7300009f50 tx=0x7f7300009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:46.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.239+0000 7f7309ffb700 1 -- 172.21.15.139:0/1891625427 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7300018070 con 0x7f730c0f5500 2021-11-20T06:02:46.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.239+0000 7f73135f1700 1 -- 172.21.15.139:0/1891625427 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c0f64f0 con 0x7f730c0f5500 2021-11-20T06:02:46.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.239+0000 7f7309ffb700 1 -- 172.21.15.139:0/1891625427 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7300013410 con 0x7f730c0f5500 2021-11-20T06:02:46.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.239+0000 7f7309ffb700 1 -- 172.21.15.139:0/1891625427 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f730001c620 con 0x7f730c0f5500 2021-11-20T06:02:46.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.239+0000 7f73135f1700 1 -- 172.21.15.139:0/1891625427 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c0f6a50 con 0x7f730c0f5500 2021-11-20T06:02:46.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.240+0000 7f73135f1700 1 -- 172.21.15.139:0/1891625427 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f730c05dcf0 con 0x7f730c0f5500 2021-11-20T06:02:46.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.240+0000 7f7309ffb700 1 -- 172.21.15.139:0/1891625427 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f730001d420 con 0x7f730c0f5500 2021-11-20T06:02:46.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.240+0000 7f7309ffb700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f72f405c2a0 0x7f72f405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:46.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.241+0000 7f7309ffb700 1 -- 172.21.15.139:0/1891625427 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7300077a40 con 0x7f730c0f5500 2021-11-20T06:02:46.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.241+0000 7f731138d700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f72f405c2a0 0x7f72f405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:46.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.241+0000 7f7309ffb700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.178:6800/927121970,v1:172.21.15.178:6801/927121970] conn(0x7f72f4062030 0x7f72f4064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:46.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.241+0000 7f7309ffb700 1 -- 172.21.15.139:0/1891625427 --> [v2:172.21.15.178:6800/927121970,v1:172.21.15.178:6801/927121970] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72f4064b10 con 0x7f72f4062030 2021-11-20T06:02:46.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.241+0000 7f7309ffb700 1 -- 172.21.15.139:0/1891625427 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f730007c220 con 0x7f730c0f5500 2021-11-20T06:02:46.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.241+0000 7f731138d700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f72f405c2a0 0x7f72f405e760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f72fc004550 tx=0x7f72fc009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:46.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.241+0000 7f7311b8e700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.178:6800/927121970,v1:172.21.15.178:6801/927121970] conn(0x7f72f4062030 0x7f72f4064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:46.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.241+0000 7f7311b8e700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.178:6800/927121970,v1:172.21.15.178:6801/927121970] conn(0x7f72f4062030 0x7f72f4064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:46.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.242+0000 7f7309ffb700 1 -- 172.21.15.139:0/1891625427 <== osd.3 v2:172.21.15.178:6800/927121970 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f72f4064b10 con 0x7f72f4062030 2021-11-20T06:02:46.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.256+0000 7f73135f1700 1 -- 172.21.15.139:0/1891625427 --> [v2:172.21.15.178:6800/927121970,v1:172.21.15.178:6801/927121970] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f730c058150 con 0x7f72f4062030 2021-11-20T06:02:46.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.256+0000 7f7309ffb700 1 -- 172.21.15.139:0/1891625427 <== osd.3 v2:172.21.15.178:6800/927121970 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f730c058150 con 0x7f72f4062030 2021-11-20T06:02:46.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.257+0000 7f73135f1700 1 -- 172.21.15.139:0/1891625427 >> [v2:172.21.15.178:6800/927121970,v1:172.21.15.178:6801/927121970] conn(0x7f72f4062030 msgr2=0x7f72f4064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:46.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.257+0000 7f73135f1700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.178:6800/927121970,v1:172.21.15.178:6801/927121970] conn(0x7f72f4062030 0x7f72f4064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:46.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.257+0000 7f73135f1700 1 -- 172.21.15.139:0/1891625427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f72f405c2a0 msgr2=0x7f72f405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:46.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.257+0000 7f73135f1700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f72f405c2a0 0x7f72f405e760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f72fc004550 tx=0x7f72fc009040).stop 2021-11-20T06:02:46.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.257+0000 7f73135f1700 1 -- 172.21.15.139:0/1891625427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f730c0f5500 msgr2=0x7f730c0f5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:46.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.257+0000 7f73135f1700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f730c0f5500 0x7f730c0f5bd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7300009f50 tx=0x7f7300009b30).stop 2021-11-20T06:02:46.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.257+0000 7f73135f1700 1 -- 172.21.15.139:0/1891625427 shutdown_connections 2021-11-20T06:02:46.246 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.257+0000 7f73135f1700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.178:6800/927121970,v1:172.21.15.178:6801/927121970] conn(0x7f72f4062030 0x7f72f4064490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:46.246 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.257+0000 7f73135f1700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f72f405c2a0 0x7f72f405e760 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:46.246 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.257+0000 7f73135f1700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f730c0f4b60 0x7f730c0f7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:46.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.257+0000 7f73135f1700 1 --2- 172.21.15.139:0/1891625427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f730c0f5500 0x7f730c0f5bd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:46.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.257+0000 7f73135f1700 1 -- 172.21.15.139:0/1891625427 >> 172.21.15.139:0/1891625427 conn(0x7f730c0f05e0 msgr2=0x7f730c0f8610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:46.248 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.257+0000 7f73135f1700 1 -- 172.21.15.139:0/1891625427 shutdown_connections 2021-11-20T06:02:46.248 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:46.257+0000 7f73135f1700 1 -- 172.21.15.139:0/1891625427 wait complete. 2021-11-20T06:02:46.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:46 smithi178 conmon[36600]: cluster 2021-11-20T06:02:44.201113+0000 mgr.smithi139.dplyxn (mgr.14162) 155 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:46.490 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:46 smithi139 conmon[33501]: cluster 2021-11-20T06:02:44.201113+0000 mgr.smithi139.dplyxn (mgr.14162) 155 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:46.666 INFO:teuthology.orchestra.run.smithi139.stdout:111669149709 2021-11-20T06:02:46.666 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2021-11-20T06:02:47.103 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:02:48.266 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.280+0000 7f0de290e700 1 -- 172.21.15.139:0/3531552863 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ddc060870 msgr2=0x7f0ddc0f93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:48.266 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.280+0000 7f0de290e700 1 --2- 172.21.15.139:0/3531552863 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ddc060870 0x7f0ddc0f93c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f0dcc004660 tx=0x7f0dcc009b30).stop 2021-11-20T06:02:48.267 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:48 smithi139 conmon[33501]: cluster 2021-11-20T06:02:46.201401+0000 mgr.smithi139.dplyxn (mgr.14162) 156 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:48.270 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.280+0000 7f0de290e700 1 -- 172.21.15.139:0/3531552863 shutdown_connections 2021-11-20T06:02:48.270 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.280+0000 7f0de290e700 1 --2- 172.21.15.139:0/3531552863 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ddc0f9940 0x7f0ddc0fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:48.271 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.280+0000 7f0de290e700 1 --2- 172.21.15.139:0/3531552863 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ddc060870 0x7f0ddc0f93c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:48.271 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.280+0000 7f0de290e700 1 -- 172.21.15.139:0/3531552863 >> 172.21.15.139:0/3531552863 conn(0x7f0ddc0f0e20 msgr2=0x7f0ddc0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:48.271 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.280+0000 7f0de290e700 1 -- 172.21.15.139:0/3531552863 shutdown_connections 2021-11-20T06:02:48.271 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.280+0000 7f0de290e700 1 -- 172.21.15.139:0/3531552863 wait complete. 2021-11-20T06:02:48.271 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.280+0000 7f0de290e700 1 Processor -- start 2021-11-20T06:02:48.272 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0de290e700 1 -- start start 2021-11-20T06:02:48.272 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0de290e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ddc060870 0x7f0ddc1070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:48.272 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0de290e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ddc0f9940 0x7f0ddc1075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:48.272 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0de290e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc107b50 con 0x7f0ddc060870 2021-11-20T06:02:48.273 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0de290e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc107c90 con 0x7f0ddc0f9940 2021-11-20T06:02:48.273 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0ddbfff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ddc060870 0x7f0ddc1070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:48.273 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0ddbfff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ddc060870 0x7f0ddc1070b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57210/0 (socket says 172.21.15.139:57210) 2021-11-20T06:02:48.273 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0ddbfff700 1 -- 172.21.15.139:0/1206514737 learned_addr learned my addr 172.21.15.139:0/1206514737 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:48.274 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0ddb7fe700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ddc0f9940 0x7f0ddc1075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:48.274 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0ddbfff700 1 -- 172.21.15.139:0/1206514737 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ddc0f9940 msgr2=0x7f0ddc1075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:48.274 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0ddbfff700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ddc0f9940 0x7f0ddc1075a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:48.275 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0ddbfff700 1 -- 172.21.15.139:0/1206514737 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc005040 con 0x7f0ddc060870 2021-11-20T06:02:48.275 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0ddb7fe700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ddc0f9940 0x7f0ddc1075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:02:48.275 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0ddbfff700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ddc060870 0x7f0ddc1070b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f0dcc009830 tx=0x7f0dcc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:48.275 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0dd8ff9700 1 -- 172.21.15.139:0/1206514737 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dcc018070 con 0x7f0ddc060870 2021-11-20T06:02:48.276 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.281+0000 7f0de290e700 1 -- 172.21.15.139:0/1206514737 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ddc1085d0 con 0x7f0ddc060870 2021-11-20T06:02:48.276 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.282+0000 7f0dd8ff9700 1 -- 172.21.15.139:0/1206514737 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0dcc013410 con 0x7f0ddc060870 2021-11-20T06:02:48.276 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.282+0000 7f0dd8ff9700 1 -- 172.21.15.139:0/1206514737 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dcc01c620 con 0x7f0ddc060870 2021-11-20T06:02:48.277 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.282+0000 7f0de290e700 1 -- 172.21.15.139:0/1206514737 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ddc108b30 con 0x7f0ddc060870 2021-11-20T06:02:48.277 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.282+0000 7f0de290e700 1 -- 172.21.15.139:0/1206514737 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0ddc05dcf0 con 0x7f0ddc060870 2021-11-20T06:02:48.277 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.283+0000 7f0dd8ff9700 1 -- 172.21.15.139:0/1206514737 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0dcc01c8e0 con 0x7f0ddc060870 2021-11-20T06:02:48.277 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.283+0000 7f0dd8ff9700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0dc405c2a0 0x7f0dc405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:48.277 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.283+0000 7f0dd8ff9700 1 -- 172.21.15.139:0/1206514737 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f0dcc077a60 con 0x7f0ddc060870 2021-11-20T06:02:48.278 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.283+0000 7f0ddb7fe700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0dc405c2a0 0x7f0dc405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:48.278 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.283+0000 7f0dd8ff9700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.178:6808/2601954583,v1:172.21.15.178:6809/2601954583] conn(0x7f0dc4062030 0x7f0dc4064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:48.278 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.283+0000 7f0dd8ff9700 1 -- 172.21.15.139:0/1206514737 --> [v2:172.21.15.178:6808/2601954583,v1:172.21.15.178:6809/2601954583] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0dc4064b10 con 0x7f0dc4062030 2021-11-20T06:02:48.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.283+0000 7f0dd8ff9700 1 -- 172.21.15.139:0/1206514737 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0dcc07c220 con 0x7f0ddc060870 2021-11-20T06:02:48.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.283+0000 7f0ddb7fe700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0dc405c2a0 0x7f0dc405e760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f0ddc060640 tx=0x7f0dd0009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:48.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.284+0000 7f0de0eab700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.178:6808/2601954583,v1:172.21.15.178:6809/2601954583] conn(0x7f0dc4062030 0x7f0dc4064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:48.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.284+0000 7f0de0eab700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.178:6808/2601954583,v1:172.21.15.178:6809/2601954583] conn(0x7f0dc4062030 0x7f0dc4064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:48.280 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.285+0000 7f0dd8ff9700 1 -- 172.21.15.139:0/1206514737 <== osd.4 v2:172.21.15.178:6808/2601954583 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f0dc4064b10 con 0x7f0dc4062030 2021-11-20T06:02:48.284 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.299+0000 7f0de290e700 1 -- 172.21.15.139:0/1206514737 --> [v2:172.21.15.178:6808/2601954583,v1:172.21.15.178:6809/2601954583] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0ddc109010 con 0x7f0dc4062030 2021-11-20T06:02:48.285 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.299+0000 7f0dd8ff9700 1 -- 172.21.15.139:0/1206514737 <== osd.4 v2:172.21.15.178:6808/2601954583 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0ddc109010 con 0x7f0dc4062030 2021-11-20T06:02:48.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.300+0000 7f0de290e700 1 -- 172.21.15.139:0/1206514737 >> [v2:172.21.15.178:6808/2601954583,v1:172.21.15.178:6809/2601954583] conn(0x7f0dc4062030 msgr2=0x7f0dc4064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:48.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.300+0000 7f0de290e700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.178:6808/2601954583,v1:172.21.15.178:6809/2601954583] conn(0x7f0dc4062030 0x7f0dc4064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:48.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.300+0000 7f0de290e700 1 -- 172.21.15.139:0/1206514737 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0dc405c2a0 msgr2=0x7f0dc405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:48.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.300+0000 7f0de290e700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0dc405c2a0 0x7f0dc405e760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f0ddc060640 tx=0x7f0dd0009040).stop 2021-11-20T06:02:48.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.300+0000 7f0de290e700 1 -- 172.21.15.139:0/1206514737 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ddc060870 msgr2=0x7f0ddc1070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:48.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.300+0000 7f0de290e700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ddc060870 0x7f0ddc1070b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f0dcc009830 tx=0x7f0dcc009b30).stop 2021-11-20T06:02:48.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.300+0000 7f0de290e700 1 -- 172.21.15.139:0/1206514737 shutdown_connections 2021-11-20T06:02:48.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.300+0000 7f0de290e700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.178:6808/2601954583,v1:172.21.15.178:6809/2601954583] conn(0x7f0dc4062030 0x7f0dc4064490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:48.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.300+0000 7f0de290e700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0dc405c2a0 0x7f0dc405e760 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:48.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.300+0000 7f0de290e700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0ddc060870 0x7f0ddc1070b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:48.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.300+0000 7f0de290e700 1 --2- 172.21.15.139:0/1206514737 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ddc0f9940 0x7f0ddc1075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:48.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.300+0000 7f0de290e700 1 -- 172.21.15.139:0/1206514737 >> 172.21.15.139:0/1206514737 conn(0x7f0ddc0f0e20 msgr2=0x7f0ddc0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:48.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.300+0000 7f0de290e700 1 -- 172.21.15.139:0/1206514737 shutdown_connections 2021-11-20T06:02:48.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:48.300+0000 7f0de290e700 1 -- 172.21.15.139:0/1206514737 wait complete. 2021-11-20T06:02:48.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:48 smithi178 conmon[36600]: cluster 2021-11-20T06:02:46.201401+0000 mgr.smithi139.dplyxn (mgr.14162) 156 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:48.711 INFO:teuthology.orchestra.run.smithi139.stdout:133143986187 2021-11-20T06:02:48.712 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2021-11-20T06:02:49.143 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:02:50.390 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.404+0000 7ff008181700 1 -- 172.21.15.139:0/4183878592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0000f59b0 msgr2=0x7ff0000f5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:50.391 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.404+0000 7ff008181700 1 --2- 172.21.15.139:0/4183878592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0000f59b0 0x7ff0000f5e30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7feff0004660 tx=0x7feff0009b30).stop 2021-11-20T06:02:50.391 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.404+0000 7ff008181700 1 -- 172.21.15.139:0/4183878592 shutdown_connections 2021-11-20T06:02:50.391 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.404+0000 7ff008181700 1 --2- 172.21.15.139:0/4183878592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0000f59b0 0x7ff0000f5e30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:50.392 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.404+0000 7ff008181700 1 --2- 172.21.15.139:0/4183878592 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff0000f6cb0 0x7ff0000f70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:50.392 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.404+0000 7ff008181700 1 -- 172.21.15.139:0/4183878592 >> 172.21.15.139:0/4183878592 conn(0x7ff0000f05e0 msgr2=0x7ff0000f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:50.392 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:50 smithi139 conmon[33501]: cluster 2021-11-20T06:02:48.201732+0000 mgr.smithi139.dplyxn (mgr.14162) 157 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:50.397 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.405+0000 7ff008181700 1 -- 172.21.15.139:0/4183878592 shutdown_connections 2021-11-20T06:02:50.397 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.405+0000 7ff008181700 1 -- 172.21.15.139:0/4183878592 wait complete. 2021-11-20T06:02:50.397 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.405+0000 7ff008181700 1 Processor -- start 2021-11-20T06:02:50.397 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.405+0000 7ff008181700 1 -- start start 2021-11-20T06:02:50.398 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7ff008181700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0000f59b0 0x7ff0000fa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:50.398 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7ff008181700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff0000f6cb0 0x7ff0000f8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:50.398 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7ff008181700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0000f8a80 con 0x7ff0000f59b0 2021-11-20T06:02:50.398 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7ff008181700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0000f8bc0 con 0x7ff0000f6cb0 2021-11-20T06:02:50.399 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7ff005f1d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0000f59b0 0x7ff0000fa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:50.399 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7ff005f1d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0000f59b0 0x7ff0000fa0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57222/0 (socket says 172.21.15.139:57222) 2021-11-20T06:02:50.399 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7ff005f1d700 1 -- 172.21.15.139:0/3147777464 learned_addr learned my addr 172.21.15.139:0/3147777464 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:50.399 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7ff00571c700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff0000f6cb0 0x7ff0000f8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:50.399 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7ff005f1d700 1 -- 172.21.15.139:0/3147777464 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff0000f6cb0 msgr2=0x7ff0000f8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:50.400 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7ff005f1d700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff0000f6cb0 0x7ff0000f8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:50.400 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7ff005f1d700 1 -- 172.21.15.139:0/3147777464 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff0005040 con 0x7ff0000f59b0 2021-11-20T06:02:50.400 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7ff005f1d700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0000f59b0 0x7ff0000fa0a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7feffc00e7c0 tx=0x7feffc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:50.400 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7feff67fc700 1 -- 172.21.15.139:0/3147777464 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feffc0092e0 con 0x7ff0000f59b0 2021-11-20T06:02:50.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7feff67fc700 1 -- 172.21.15.139:0/3147777464 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feffc00f040 con 0x7ff0000f59b0 2021-11-20T06:02:50.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7ff008181700 1 -- 172.21.15.139:0/3147777464 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0000f8ea0 con 0x7ff0000f59b0 2021-11-20T06:02:50.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7feff67fc700 1 -- 172.21.15.139:0/3147777464 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feffc0147c0 con 0x7ff0000f59b0 2021-11-20T06:02:50.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.406+0000 7ff008181700 1 -- 172.21.15.139:0/3147777464 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff00010f0e0 con 0x7ff0000f59b0 2021-11-20T06:02:50.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.407+0000 7ff008181700 1 -- 172.21.15.139:0/3147777464 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff00005dcf0 con 0x7ff0000f59b0 2021-11-20T06:02:50.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.408+0000 7feff67fc700 1 -- 172.21.15.139:0/3147777464 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7feffc01b030 con 0x7ff0000f59b0 2021-11-20T06:02:50.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.408+0000 7feff67fc700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fefec05c2a0 0x7fefec05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:50.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.408+0000 7feff67fc700 1 -- 172.21.15.139:0/3147777464 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7feffc075a50 con 0x7ff0000f59b0 2021-11-20T06:02:50.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.408+0000 7ff00571c700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fefec05c2a0 0x7fefec05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:50.403 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.408+0000 7feff67fc700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.178:6816/3932451665,v1:172.21.15.178:6817/3932451665] conn(0x7fefec062030 0x7fefec064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:50.403 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.408+0000 7feff67fc700 1 -- 172.21.15.139:0/3147777464 --> [v2:172.21.15.178:6816/3932451665,v1:172.21.15.178:6817/3932451665] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefec064b10 con 0x7fefec062030 2021-11-20T06:02:50.403 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.408+0000 7feff67fc700 1 -- 172.21.15.139:0/3147777464 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7feffc07a220 con 0x7ff0000f59b0 2021-11-20T06:02:50.403 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.408+0000 7ff00571c700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fefec05c2a0 0x7fefec05e760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7feff0013530 tx=0x7feff0016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:50.404 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.409+0000 7ff00671e700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.178:6816/3932451665,v1:172.21.15.178:6817/3932451665] conn(0x7fefec062030 0x7fefec064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:50.404 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.409+0000 7ff00671e700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.178:6816/3932451665,v1:172.21.15.178:6817/3932451665] conn(0x7fefec062030 0x7fefec064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:50.404 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.410+0000 7feff67fc700 1 -- 172.21.15.139:0/3147777464 <== osd.5 v2:172.21.15.178:6816/3932451665 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fefec064b10 con 0x7fefec062030 2021-11-20T06:02:50.409 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.424+0000 7ff008181700 1 -- 172.21.15.139:0/3147777464 --> [v2:172.21.15.178:6816/3932451665,v1:172.21.15.178:6817/3932451665] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff0000f51f0 con 0x7fefec062030 2021-11-20T06:02:50.410 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.425+0000 7feff67fc700 1 -- 172.21.15.139:0/3147777464 <== osd.5 v2:172.21.15.178:6816/3932451665 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff0000f51f0 con 0x7fefec062030 2021-11-20T06:02:50.411 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.425+0000 7ff008181700 1 -- 172.21.15.139:0/3147777464 >> [v2:172.21.15.178:6816/3932451665,v1:172.21.15.178:6817/3932451665] conn(0x7fefec062030 msgr2=0x7fefec064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:50.411 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.425+0000 7ff008181700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.178:6816/3932451665,v1:172.21.15.178:6817/3932451665] conn(0x7fefec062030 0x7fefec064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:50.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.425+0000 7ff008181700 1 -- 172.21.15.139:0/3147777464 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fefec05c2a0 msgr2=0x7fefec05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:50.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.425+0000 7ff008181700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fefec05c2a0 0x7fefec05e760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7feff0013530 tx=0x7feff0016040).stop 2021-11-20T06:02:50.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.425+0000 7ff008181700 1 -- 172.21.15.139:0/3147777464 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0000f59b0 msgr2=0x7ff0000fa0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:50.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.425+0000 7ff008181700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0000f59b0 0x7ff0000fa0a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7feffc00e7c0 tx=0x7feffc00b040).stop 2021-11-20T06:02:50.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.425+0000 7ff008181700 1 -- 172.21.15.139:0/3147777464 shutdown_connections 2021-11-20T06:02:50.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.425+0000 7ff008181700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fefec05c2a0 0x7fefec05e760 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:50.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.425+0000 7ff008181700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.178:6816/3932451665,v1:172.21.15.178:6817/3932451665] conn(0x7fefec062030 0x7fefec064490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:50.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.425+0000 7ff008181700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0000f59b0 0x7ff0000fa0a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:50.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.425+0000 7ff008181700 1 --2- 172.21.15.139:0/3147777464 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff0000f6cb0 0x7ff0000f8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:50.414 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.425+0000 7ff008181700 1 -- 172.21.15.139:0/3147777464 >> 172.21.15.139:0/3147777464 conn(0x7ff0000f05e0 msgr2=0x7ff0000f48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:50.414 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.425+0000 7ff008181700 1 -- 172.21.15.139:0/3147777464 shutdown_connections 2021-11-20T06:02:50.414 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:50.425+0000 7ff008181700 1 -- 172.21.15.139:0/3147777464 wait complete. 2021-11-20T06:02:50.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:50 smithi178 conmon[36600]: cluster 2021-11-20T06:02:48.201732+0000 mgr.smithi139.dplyxn (mgr.14162) 157 : cluster [DBG] 2021-11-20T06:02:50.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:50 smithi178 conmon[36600]: pgmap v126: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:50.853 INFO:teuthology.orchestra.run.smithi139.stdout:150323855368 2021-11-20T06:02:50.854 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-11-20T06:02:51.288 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:02:52.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:52 smithi178 conmon[36600]: cluster 2021-11-20T06:02:50.202122+0000 2021-11-20T06:02:52.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:52 smithi178 conmon[36600]: mgr.smithi139.dplyxn (mgr.14162) 158 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:52.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:52 smithi139 conmon[33501]: cluster 2021-11-20T06:02:50.202122+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:02:52.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:52 smithi139 conmon[33501]: ) 158 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:52.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.563+0000 7f07e5cf5700 1 -- 172.21.15.139:0/1933119626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e00f5440 msgr2=0x7f07e00f7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:52.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.563+0000 7f07e5cf5700 1 --2- 172.21.15.139:0/1933119626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e00f5440 0x7f07e00f7bc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f07d4004660 tx=0x7f07d4009b30).stop 2021-11-20T06:02:52.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.563+0000 7f07e5cf5700 1 -- 172.21.15.139:0/1933119626 shutdown_connections 2021-11-20T06:02:52.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.563+0000 7f07e5cf5700 1 --2- 172.21.15.139:0/1933119626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e00f5440 0x7f07e00f7bc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:52.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.563+0000 7f07e5cf5700 1 --2- 172.21.15.139:0/1933119626 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07e00f4aa0 0x7f07e00f4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:52.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.563+0000 7f07e5cf5700 1 -- 172.21.15.139:0/1933119626 >> 172.21.15.139:0/1933119626 conn(0x7f07e00f05e0 msgr2=0x7f07e00f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:52.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.563+0000 7f07e5cf5700 1 -- 172.21.15.139:0/1933119626 shutdown_connections 2021-11-20T06:02:52.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.563+0000 7f07e5cf5700 1 -- 172.21.15.139:0/1933119626 wait complete. 2021-11-20T06:02:52.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.563+0000 7f07e5cf5700 1 Processor -- start 2021-11-20T06:02:52.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07e5cf5700 1 -- start start 2021-11-20T06:02:52.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07e5cf5700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07e00f4aa0 0x7f07e01070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:52.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07e5cf5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e00f5440 0x7f07e01075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:52.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07e5cf5700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e0107b70 con 0x7f07e00f5440 2021-11-20T06:02:52.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07e5cf5700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e0107cb0 con 0x7f07e00f4aa0 2021-11-20T06:02:52.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07deffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e00f5440 0x7f07e01075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:52.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07deffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e00f5440 0x7f07e01075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57236/0 (socket says 172.21.15.139:57236) 2021-11-20T06:02:52.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07deffd700 1 -- 172.21.15.139:0/3341559465 learned_addr learned my addr 172.21.15.139:0/3341559465 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:52.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07df7fe700 1 --2- 172.21.15.139:0/3341559465 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07e00f4aa0 0x7f07e01070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:52.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07deffd700 1 -- 172.21.15.139:0/3341559465 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07e00f4aa0 msgr2=0x7f07e01070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:52.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07deffd700 1 --2- 172.21.15.139:0/3341559465 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07e00f4aa0 0x7f07e01070d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:52.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07deffd700 1 -- 172.21.15.139:0/3341559465 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d4005040 con 0x7f07e00f5440 2021-11-20T06:02:52.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07deffd700 1 --2- 172.21.15.139:0/3341559465 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e00f5440 0x7f07e01075c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f07d4009f50 tx=0x7f07d4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:52.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07c7fff700 1 -- 172.21.15.139:0/3341559465 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07d4018070 con 0x7f07e00f5440 2021-11-20T06:02:52.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07e5cf5700 1 -- 172.21.15.139:0/3341559465 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e01085f0 con 0x7f07e00f5440 2021-11-20T06:02:52.556 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.564+0000 7f07e5cf5700 1 -- 172.21.15.139:0/3341559465 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e0108b50 con 0x7f07e00f5440 2021-11-20T06:02:52.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.565+0000 7f07c7fff700 1 -- 172.21.15.139:0/3341559465 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07d4013410 con 0x7f07e00f5440 2021-11-20T06:02:52.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.565+0000 7f07c7fff700 1 -- 172.21.15.139:0/3341559465 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07d401c3c0 con 0x7f07e00f5440 2021-11-20T06:02:52.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.565+0000 7f07e5cf5700 1 -- 172.21.15.139:0/3341559465 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c0004fa0 con 0x7f07e00f5440 2021-11-20T06:02:52.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.566+0000 7f07c7fff700 1 -- 172.21.15.139:0/3341559465 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f07d401d420 con 0x7f07e00f5440 2021-11-20T06:02:52.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.566+0000 7f07c7fff700 1 --2- 172.21.15.139:0/3341559465 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f07c805c2a0 0x7f07c805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:52.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.566+0000 7f07c7fff700 1 -- 172.21.15.139:0/3341559465 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f07d4077760 con 0x7f07e00f5440 2021-11-20T06:02:52.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.566+0000 7f07df7fe700 1 --2- 172.21.15.139:0/3341559465 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f07c805c2a0 0x7f07c805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:52.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.566+0000 7f07df7fe700 1 --2- 172.21.15.139:0/3341559465 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f07c805c2a0 0x7f07c805e760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f07e005f8e0 tx=0x7f07d0009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:52.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.570+0000 7f07c7fff700 1 -- 172.21.15.139:0/3341559465 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07d404a760 con 0x7f07e00f5440 2021-11-20T06:02:52.706 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.720+0000 7f07e5cf5700 1 -- 172.21.15.139:0/3341559465 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f07c0005670 con 0x7f07e00f5440 2021-11-20T06:02:52.706 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.721+0000 7f07c7fff700 1 -- 172.21.15.139:0/3341559465 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f07d404dd80 con 0x7f07e00f5440 2021-11-20T06:02:52.707 INFO:teuthology.orchestra.run.smithi139.stdout:38654705688 2021-11-20T06:02:52.708 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.722+0000 7f07e5cf5700 1 -- 172.21.15.139:0/3341559465 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f07c805c2a0 msgr2=0x7f07c805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:52.708 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.722+0000 7f07e5cf5700 1 --2- 172.21.15.139:0/3341559465 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f07c805c2a0 0x7f07c805e760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f07e005f8e0 tx=0x7f07d0009040).stop 2021-11-20T06:02:52.708 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.722+0000 7f07e5cf5700 1 -- 172.21.15.139:0/3341559465 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e00f5440 msgr2=0x7f07e01075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:52.709 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.722+0000 7f07e5cf5700 1 --2- 172.21.15.139:0/3341559465 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e00f5440 0x7f07e01075c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f07d4009f50 tx=0x7f07d4009b30).stop 2021-11-20T06:02:52.709 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.722+0000 7f07e5cf5700 1 -- 172.21.15.139:0/3341559465 shutdown_connections 2021-11-20T06:02:52.709 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.722+0000 7f07e5cf5700 1 --2- 172.21.15.139:0/3341559465 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f07c805c2a0 0x7f07c805e760 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:52.709 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.722+0000 7f07e5cf5700 1 --2- 172.21.15.139:0/3341559465 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07e00f4aa0 0x7f07e01070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:52.710 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.722+0000 7f07e5cf5700 1 --2- 172.21.15.139:0/3341559465 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f07e00f5440 0x7f07e01075c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:52.710 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.722+0000 7f07e5cf5700 1 -- 172.21.15.139:0/3341559465 >> 172.21.15.139:0/3341559465 conn(0x7f07e00f05e0 msgr2=0x7f07e00fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:52.710 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.722+0000 7f07e5cf5700 1 -- 172.21.15.139:0/3341559465 shutdown_connections 2021-11-20T06:02:52.710 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:52.722+0000 7f07e5cf5700 1 -- 172.21.15.139:0/3341559465 wait complete. 2021-11-20T06:02:53.084 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705686 got 38654705688 for osd.0 2021-11-20T06:02:53.085 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.1 2021-11-20T06:02:53.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:53 smithi178 conmon[36600]: cluster 2021-11-20T06:02:52.202425+0000 mgr.smithi139.dplyxn (mgr.14162) 159 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:53.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:53 smithi178 conmon[36600]: audit 2021-11-20T06:02:52.721543+0000 mon.smithi139 (mon.0) 437 : audit [DBG] from='client.? 172.21.15.139:0/3341559465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-11-20T06:02:53.521 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:02:53.688 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:53 smithi139 conmon[33501]: cluster 2021-11-20T06:02:52.202425+0000 mgr.smithi139.dplyxn (mgr.14162) 159 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:53.688 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:53 smithi139 conmon[33501]: audit 2021-11-20T06:02:52.721543+0000 mon.smithi139 (mon.0) 437 : audit [DBG] from='client.? 172.21.15.139:0/3341559465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-11-20T06:02:54.772 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.784+0000 7f154efb3700 1 -- 172.21.15.139:0/2336916943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15480f67e0 msgr2=0x7f15480f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:54.773 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.784+0000 7f154efb3700 1 --2- 172.21.15.139:0/2336916943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15480f67e0 0x7f15480f6c00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f153c004660 tx=0x7f153c009b30).stop 2021-11-20T06:02:54.773 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.784+0000 7f154efb3700 1 -- 172.21.15.139:0/2336916943 shutdown_connections 2021-11-20T06:02:54.773 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.784+0000 7f154efb3700 1 --2- 172.21.15.139:0/2336916943 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15480f7a80 0x7f15480f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:02:54.773 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.784+0000 7f154efb3700 1 --2- 172.21.15.139:0/2336916943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15480f67e0 0x7f15480f6c00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:54.774 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.784+0000 7f154efb3700 1 -- 172.21.15.139:0/2336916943 >> 172.21.15.139:0/2336916943 conn(0x7f15480f1f20 msgr2=0x7f15480f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:54.774 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.784+0000 7f154efb3700 1 -- 172.21.15.139:0/2336916943 shutdown_connections 2021-11-20T06:02:54.774 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.784+0000 7f154efb3700 1 -- 172.21.15.139:0/2336916943 wait complete. 2021-11-20T06:02:54.774 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.785+0000 7f154efb3700 1 Processor -- start 2021-11-20T06:02:54.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.785+0000 7f154efb3700 1 -- start start 2021-11-20T06:02:54.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.785+0000 7f154efb3700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15480f67e0 0x7f154806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:54.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.785+0000 7f154efb3700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15480f7a80 0x7f1548069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:54.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.785+0000 7f154efb3700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154806a410 con 0x7f15480f67e0 2021-11-20T06:02:54.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.785+0000 7f154efb3700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154806a550 con 0x7f15480f7a80 2021-11-20T06:02:54.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.785+0000 7f154cd4f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15480f67e0 0x7f154806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:54.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.785+0000 7f154cd4f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15480f67e0 0x7f154806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57244/0 (socket says 172.21.15.139:57244) 2021-11-20T06:02:54.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.785+0000 7f154cd4f700 1 -- 172.21.15.139:0/1218549450 learned_addr learned my addr 172.21.15.139:0/1218549450 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:54.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.786+0000 7f1547fff700 1 --2- 172.21.15.139:0/1218549450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15480f7a80 0x7f1548069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:54.777 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.786+0000 7f154cd4f700 1 -- 172.21.15.139:0/1218549450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15480f7a80 msgr2=0x7f1548069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:54.777 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.786+0000 7f154cd4f700 1 --2- 172.21.15.139:0/1218549450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15480f7a80 0x7f1548069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:54.777 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.786+0000 7f154cd4f700 1 -- 172.21.15.139:0/1218549450 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c005040 con 0x7f15480f67e0 2021-11-20T06:02:54.777 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.786+0000 7f1547fff700 1 --2- 172.21.15.139:0/1218549450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15480f7a80 0x7f1548069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:02:54.778 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.786+0000 7f154cd4f700 1 --2- 172.21.15.139:0/1218549450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15480f67e0 0x7f154806b9a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f153c009f50 tx=0x7f153c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:54.778 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.786+0000 7f15457fa700 1 -- 172.21.15.139:0/1218549450 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f153c018070 con 0x7f15480f67e0 2021-11-20T06:02:54.778 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.786+0000 7f15457fa700 1 -- 172.21.15.139:0/1218549450 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f153c013410 con 0x7f15480f67e0 2021-11-20T06:02:54.778 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.786+0000 7f154efb3700 1 -- 172.21.15.139:0/1218549450 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154806a7d0 con 0x7f15480f67e0 2021-11-20T06:02:54.778 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.786+0000 7f154efb3700 1 -- 172.21.15.139:0/1218549450 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154810f130 con 0x7f15480f67e0 2021-11-20T06:02:54.779 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.786+0000 7f15457fa700 1 -- 172.21.15.139:0/1218549450 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f153c01c620 con 0x7f15480f67e0 2021-11-20T06:02:54.779 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.787+0000 7f154efb3700 1 -- 172.21.15.139:0/1218549450 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1528004fa0 con 0x7f15480f67e0 2021-11-20T06:02:54.779 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.787+0000 7f15457fa700 1 -- 172.21.15.139:0/1218549450 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f153c01c890 con 0x7f15480f67e0 2021-11-20T06:02:54.780 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.788+0000 7f15457fa700 1 --2- 172.21.15.139:0/1218549450 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f153805c2a0 0x7f153805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:54.781 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.788+0000 7f15457fa700 1 -- 172.21.15.139:0/1218549450 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f153c0779c0 con 0x7f15480f67e0 2021-11-20T06:02:54.781 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.788+0000 7f1547fff700 1 --2- 172.21.15.139:0/1218549450 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f153805c2a0 0x7f153805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:54.781 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.788+0000 7f1547fff700 1 --2- 172.21.15.139:0/1218549450 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f153805c2a0 0x7f153805e760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f153400a880 tx=0x7f1534008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:54.781 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.792+0000 7f15457fa700 1 -- 172.21.15.139:0/1218549450 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f153c04e2a0 con 0x7f15480f67e0 2021-11-20T06:02:54.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.940+0000 7f154efb3700 1 -- 172.21.15.139:0/1218549450 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1528005670 con 0x7f15480f67e0 2021-11-20T06:02:54.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.941+0000 7f15457fa700 1 -- 172.21.15.139:0/1218549450 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f153c04e4b0 con 0x7f15480f67e0 2021-11-20T06:02:54.927 INFO:teuthology.orchestra.run.smithi139.stdout:60129542166 2021-11-20T06:02:54.928 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.942+0000 7f154efb3700 1 -- 172.21.15.139:0/1218549450 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f153805c2a0 msgr2=0x7f153805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:54.928 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.942+0000 7f154efb3700 1 --2- 172.21.15.139:0/1218549450 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f153805c2a0 0x7f153805e760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f153400a880 tx=0x7f1534008040).stop 2021-11-20T06:02:54.929 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.942+0000 7f154efb3700 1 -- 172.21.15.139:0/1218549450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15480f67e0 msgr2=0x7f154806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:54.929 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.942+0000 7f154efb3700 1 --2- 172.21.15.139:0/1218549450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15480f67e0 0x7f154806b9a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f153c009f50 tx=0x7f153c009b30).stop 2021-11-20T06:02:54.929 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.942+0000 7f154efb3700 1 -- 172.21.15.139:0/1218549450 shutdown_connections 2021-11-20T06:02:54.929 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.942+0000 7f154efb3700 1 --2- 172.21.15.139:0/1218549450 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f153805c2a0 0x7f153805e760 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:54.930 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.942+0000 7f154efb3700 1 --2- 172.21.15.139:0/1218549450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15480f67e0 0x7f154806b9a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:54.930 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.942+0000 7f154efb3700 1 --2- 172.21.15.139:0/1218549450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15480f7a80 0x7f1548069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:54.930 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.942+0000 7f154efb3700 1 -- 172.21.15.139:0/1218549450 >> 172.21.15.139:0/1218549450 conn(0x7f15480f1f20 msgr2=0x7f15480fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:54.930 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.942+0000 7f154efb3700 1 -- 172.21.15.139:0/1218549450 shutdown_connections 2021-11-20T06:02:54.930 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:54.943+0000 7f154efb3700 1 -- 172.21.15.139:0/1218549450 wait complete. 2021-11-20T06:02:55.319 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542164 got 60129542166 for osd.1 2021-11-20T06:02:55.320 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.2 2021-11-20T06:02:55.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:55 smithi139 conmon[33501]: cluster 2021-11-20T06:02:54.202843+0000 mgr.smithi139.dplyxn (mgr.14162) 160 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:55.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:55 smithi139 conmon[33501]: audit 2021-11-20T06:02:54.941697+0000 mon.smithi139 (mon.0) 438 : audit [DBG] from='client.? 172.21.15.139:0/1218549450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-11-20T06:02:55.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:55 smithi178 conmon[36600]: cluster 2021-11-20T06:02:54.202843+0000 mgr.smithi139.dplyxn (mgr.14162) 160 : cluster 2021-11-20T06:02:55.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:55 smithi178 conmon[36600]: [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:55.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:55 smithi178 conmon[36600]: audit 2021-11-20T06:02:54.941697+0000 mon.smithi139 (mon.0) 438 : audit [DBG] from='client.? 172.21.15.139:0/1218549450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-11-20T06:02:55.755 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:02:57.011 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.023+0000 7fbbf9b17700 1 -- 172.21.15.139:0/3255363405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf40f7a80 msgr2=0x7fbbf40f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:57.011 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.023+0000 7fbbf9b17700 1 --2- 172.21.15.139:0/3255363405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf40f7a80 0x7fbbf40f7ee0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fbbe8004660 tx=0x7fbbe8009b30).stop 2021-11-20T06:02:57.011 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.023+0000 7fbbf9b17700 1 -- 172.21.15.139:0/3255363405 shutdown_connections 2021-11-20T06:02:57.012 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.023+0000 7fbbf9b17700 1 --2- 172.21.15.139:0/3255363405 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf40f7a80 0x7fbbf40f7ee0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:57.012 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.023+0000 7fbbf9b17700 1 --2- 172.21.15.139:0/3255363405 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf40f67e0 0x7fbbf40f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:57.012 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.023+0000 7fbbf9b17700 1 -- 172.21.15.139:0/3255363405 >> 172.21.15.139:0/3255363405 conn(0x7fbbf40f1f20 msgr2=0x7fbbf40f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:57.012 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.023+0000 7fbbf9b17700 1 -- 172.21.15.139:0/3255363405 shutdown_connections 2021-11-20T06:02:57.013 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.023+0000 7fbbf9b17700 1 -- 172.21.15.139:0/3255363405 wait complete. 2021-11-20T06:02:57.013 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.023+0000 7fbbf9b17700 1 Processor -- start 2021-11-20T06:02:57.013 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.024+0000 7fbbf9b17700 1 -- start start 2021-11-20T06:02:57.014 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.024+0000 7fbbf9b17700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf40f67e0 0x7fbbf4107050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:57.014 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.024+0000 7fbbf9b17700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf40f7a80 0x7fbbf4107540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:57.015 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.024+0000 7fbbf9b17700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf4107af0 con 0x7fbbf40f7a80 2021-11-20T06:02:57.015 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.024+0000 7fbbf9b17700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf4107c30 con 0x7fbbf40f67e0 2021-11-20T06:02:57.015 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.024+0000 7fbbf2ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf40f7a80 0x7fbbf4107540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:57.015 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.024+0000 7fbbf2ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf40f7a80 0x7fbbf4107540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57254/0 (socket says 172.21.15.139:57254) 2021-11-20T06:02:57.016 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.024+0000 7fbbf2ffd700 1 -- 172.21.15.139:0/3127210507 learned_addr learned my addr 172.21.15.139:0/3127210507 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:57.016 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.024+0000 7fbbf2ffd700 1 -- 172.21.15.139:0/3127210507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf40f67e0 msgr2=0x7fbbf4107050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:57.016 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.024+0000 7fbbf37fe700 1 --2- 172.21.15.139:0/3127210507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf40f67e0 0x7fbbf4107050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:57.017 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.024+0000 7fbbf2ffd700 1 --2- 172.21.15.139:0/3127210507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf40f67e0 0x7fbbf4107050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:57.017 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.024+0000 7fbbf2ffd700 1 -- 172.21.15.139:0/3127210507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe8005040 con 0x7fbbf40f7a80 2021-11-20T06:02:57.017 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.024+0000 7fbbf37fe700 1 --2- 172.21.15.139:0/3127210507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf40f67e0 0x7fbbf4107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:02:57.018 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.025+0000 7fbbf2ffd700 1 --2- 172.21.15.139:0/3127210507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf40f7a80 0x7fbbf4107540 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fbbe8009830 tx=0x7fbbe8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:57.018 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.025+0000 7fbbdbfff700 1 -- 172.21.15.139:0/3127210507 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbe8018070 con 0x7fbbf40f7a80 2021-11-20T06:02:57.018 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.025+0000 7fbbf9b17700 1 -- 172.21.15.139:0/3127210507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf4108570 con 0x7fbbf40f7a80 2021-11-20T06:02:57.019 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.025+0000 7fbbdbfff700 1 -- 172.21.15.139:0/3127210507 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbe8013410 con 0x7fbbf40f7a80 2021-11-20T06:02:57.019 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.025+0000 7fbbdbfff700 1 -- 172.21.15.139:0/3127210507 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbe801c620 con 0x7fbbf40f7a80 2021-11-20T06:02:57.020 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.025+0000 7fbbf9b17700 1 -- 172.21.15.139:0/3127210507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf4108ad0 con 0x7fbbf40f7a80 2021-11-20T06:02:57.021 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.026+0000 7fbbf9b17700 1 -- 172.21.15.139:0/3127210507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf4045ad0 con 0x7fbbf40f7a80 2021-11-20T06:02:57.022 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.026+0000 7fbbdbfff700 1 -- 172.21.15.139:0/3127210507 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbbe801c8e0 con 0x7fbbf40f7a80 2021-11-20T06:02:57.022 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.026+0000 7fbbdbfff700 1 --2- 172.21.15.139:0/3127210507 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fbbdc05c2a0 0x7fbbdc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:57.022 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.026+0000 7fbbdbfff700 1 -- 172.21.15.139:0/3127210507 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fbbe8077a60 con 0x7fbbf40f7a80 2021-11-20T06:02:57.022 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.026+0000 7fbbf37fe700 1 --2- 172.21.15.139:0/3127210507 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fbbdc05c2a0 0x7fbbdc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:57.022 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.027+0000 7fbbf37fe700 1 --2- 172.21.15.139:0/3127210507 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fbbdc05c2a0 0x7fbbdc05e760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fbbe4004fd0 tx=0x7fbbe4009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:57.023 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.030+0000 7fbbdbfff700 1 -- 172.21.15.139:0/3127210507 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbbe804aa00 con 0x7fbbf40f7a80 2021-11-20T06:02:57.166 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.180+0000 7fbbf9b17700 1 -- 172.21.15.139:0/3127210507 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fbbf4027a00 con 0x7fbbf40f7a80 2021-11-20T06:02:57.167 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.180+0000 7fbbdbfff700 1 -- 172.21.15.139:0/3127210507 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbbe804e020 con 0x7fbbf40f7a80 2021-11-20T06:02:57.167 INFO:teuthology.orchestra.run.smithi139.stdout:77309411346 2021-11-20T06:02:57.168 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.182+0000 7fbbf9b17700 1 -- 172.21.15.139:0/3127210507 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fbbdc05c2a0 msgr2=0x7fbbdc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:57.168 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.182+0000 7fbbf9b17700 1 --2- 172.21.15.139:0/3127210507 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fbbdc05c2a0 0x7fbbdc05e760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fbbe4004fd0 tx=0x7fbbe4009040).stop 2021-11-20T06:02:57.168 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.182+0000 7fbbf9b17700 1 -- 172.21.15.139:0/3127210507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf40f7a80 msgr2=0x7fbbf4107540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:57.169 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.182+0000 7fbbf9b17700 1 --2- 172.21.15.139:0/3127210507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf40f7a80 0x7fbbf4107540 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fbbe8009830 tx=0x7fbbe8009b30).stop 2021-11-20T06:02:57.169 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.182+0000 7fbbf9b17700 1 -- 172.21.15.139:0/3127210507 shutdown_connections 2021-11-20T06:02:57.169 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.182+0000 7fbbf9b17700 1 --2- 172.21.15.139:0/3127210507 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fbbdc05c2a0 0x7fbbdc05e760 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:57.169 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.182+0000 7fbbf9b17700 1 --2- 172.21.15.139:0/3127210507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf40f67e0 0x7fbbf4107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:57.170 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.182+0000 7fbbf9b17700 1 --2- 172.21.15.139:0/3127210507 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf40f7a80 0x7fbbf4107540 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:57.170 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.182+0000 7fbbf9b17700 1 -- 172.21.15.139:0/3127210507 >> 172.21.15.139:0/3127210507 conn(0x7fbbf40f1f20 msgr2=0x7fbbf40fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:57.170 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.182+0000 7fbbf9b17700 1 -- 172.21.15.139:0/3127210507 shutdown_connections 2021-11-20T06:02:57.170 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:57.182+0000 7fbbf9b17700 1 -- 172.21.15.139:0/3127210507 wait complete. 2021-11-20T06:02:57.553 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411343 got 77309411346 for osd.2 2021-11-20T06:02:57.554 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.3 2021-11-20T06:02:57.989 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:02:58.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:58 smithi178 conmon[36600]: cluster 2021-11-20T06:02:56.203269+0000 mgr.smithi139.dplyxn (mgr.14162) 161 : cluster 2021-11-20T06:02:58.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:58 smithi178 conmon[36600]: [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:58.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:02:58 smithi178 conmon[36600]: audit 2021-11-20T06:02:57.181391+0000 mon.smithi139 (mon.0) 439 : audit [DBG] from='client.? 172.21.15.139:0/3127210507' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-11-20T06:02:58.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:58 smithi139 conmon[33501]: cluster 2021-11-20T06:02:56.203269+0000 mgr.smithi139.dplyxn (mgr.14162) 161 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:02:58.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:02:58 smithi139 conmon[33501]: audit 2021-11-20T06:02:57.181391+0000 mon.smithi139 (mon.0) 439 : audit [DBG] from='client.? 172.21.15.139:0/3127210507' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-11-20T06:02:59.211 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.225+0000 7ff021630700 1 -- 172.21.15.139:0/2002458486 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff01c0f5510 msgr2=0x7ff01c0f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:59.211 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.225+0000 7ff021630700 1 --2- 172.21.15.139:0/2002458486 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff01c0f5510 0x7ff01c0f5930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7ff00c004660 tx=0x7ff00c009b30).stop 2021-11-20T06:02:59.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.225+0000 7ff021630700 1 -- 172.21.15.139:0/2002458486 shutdown_connections 2021-11-20T06:02:59.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.225+0000 7ff021630700 1 --2- 172.21.15.139:0/2002458486 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff01c0f67b0 0x7ff01c0f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:59.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.225+0000 7ff021630700 1 --2- 172.21.15.139:0/2002458486 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff01c0f5510 0x7ff01c0f5930 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:59.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.225+0000 7ff021630700 1 -- 172.21.15.139:0/2002458486 >> 172.21.15.139:0/2002458486 conn(0x7ff01c0f0c70 msgr2=0x7ff01c0f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:59.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.225+0000 7ff021630700 1 -- 172.21.15.139:0/2002458486 shutdown_connections 2021-11-20T06:02:59.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.225+0000 7ff021630700 1 -- 172.21.15.139:0/2002458486 wait complete. 2021-11-20T06:02:59.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.226+0000 7ff021630700 1 Processor -- start 2021-11-20T06:02:59.215 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.226+0000 7ff021630700 1 -- start start 2021-11-20T06:02:59.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.226+0000 7ff021630700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff01c0f5510 0x7ff01c109bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:59.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.226+0000 7ff021630700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff01c0f67b0 0x7ff01c10c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:59.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.226+0000 7ff021630700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c060980 con 0x7ff01c0f67b0 2021-11-20T06:02:59.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.226+0000 7ff021630700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c060ac0 con 0x7ff01c0f5510 2021-11-20T06:02:59.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.227+0000 7ff01b7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff01c0f67b0 0x7ff01c10c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:59.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.227+0000 7ff01b7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff01c0f67b0 0x7ff01c10c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57264/0 (socket says 172.21.15.139:57264) 2021-11-20T06:02:59.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.227+0000 7ff01b7fe700 1 -- 172.21.15.139:0/2067117302 learned_addr learned my addr 172.21.15.139:0/2067117302 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:02:59.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.227+0000 7ff01bfff700 1 --2- 172.21.15.139:0/2067117302 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff01c0f5510 0x7ff01c109bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:59.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.227+0000 7ff01b7fe700 1 -- 172.21.15.139:0/2067117302 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff01c0f5510 msgr2=0x7ff01c109bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:59.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.227+0000 7ff01b7fe700 1 --2- 172.21.15.139:0/2067117302 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff01c0f5510 0x7ff01c109bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:59.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.227+0000 7ff01b7fe700 1 -- 172.21.15.139:0/2067117302 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c005040 con 0x7ff01c0f67b0 2021-11-20T06:02:59.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.227+0000 7ff01b7fe700 1 --2- 172.21.15.139:0/2067117302 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff01c0f67b0 0x7ff01c10c0c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff0100026d0 tx=0x7ff010009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:59.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.227+0000 7ff018ff9700 1 -- 172.21.15.139:0/2067117302 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0100079c0 con 0x7ff01c0f67b0 2021-11-20T06:02:59.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.227+0000 7ff018ff9700 1 -- 172.21.15.139:0/2067117302 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff01000f040 con 0x7ff01c0f67b0 2021-11-20T06:02:59.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.227+0000 7ff021630700 1 -- 172.21.15.139:0/2067117302 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff01c10c6d0 con 0x7ff01c0f67b0 2021-11-20T06:02:59.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.227+0000 7ff018ff9700 1 -- 172.21.15.139:0/2067117302 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0100148b0 con 0x7ff01c0f67b0 2021-11-20T06:02:59.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.227+0000 7ff021630700 1 -- 172.21.15.139:0/2067117302 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff01c10cc60 con 0x7ff01c0f67b0 2021-11-20T06:02:59.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.228+0000 7ff021630700 1 -- 172.21.15.139:0/2067117302 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff01c059100 con 0x7ff01c0f67b0 2021-11-20T06:02:59.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.228+0000 7ff018ff9700 1 -- 172.21.15.139:0/2067117302 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff010014a10 con 0x7ff01c0f67b0 2021-11-20T06:02:59.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.229+0000 7ff018ff9700 1 --2- 172.21.15.139:0/2067117302 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff00405c2b0 0x7ff00405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:02:59.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.229+0000 7ff018ff9700 1 -- 172.21.15.139:0/2067117302 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff0100759f0 con 0x7ff01c0f67b0 2021-11-20T06:02:59.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.229+0000 7ff01bfff700 1 --2- 172.21.15.139:0/2067117302 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff00405c2b0 0x7ff00405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:02:59.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.229+0000 7ff01bfff700 1 --2- 172.21.15.139:0/2067117302 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff00405c2b0 0x7ff00405e770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7ff01c060600 tx=0x7ff00c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:02:59.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.233+0000 7ff018ff9700 1 -- 172.21.15.139:0/2067117302 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff010077e60 con 0x7ff01c0f67b0 2021-11-20T06:02:59.369 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.384+0000 7ff021630700 1 -- 172.21.15.139:0/2067117302 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff01c045ad0 con 0x7ff01c0f67b0 2021-11-20T06:02:59.370 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.384+0000 7ff018ff9700 1 -- 172.21.15.139:0/2067117302 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff010048a70 con 0x7ff01c0f67b0 2021-11-20T06:02:59.370 INFO:teuthology.orchestra.run.smithi139.stdout:111669149711 2021-11-20T06:02:59.372 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.386+0000 7ff021630700 1 -- 172.21.15.139:0/2067117302 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff00405c2b0 msgr2=0x7ff00405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:59.372 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.386+0000 7ff021630700 1 --2- 172.21.15.139:0/2067117302 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff00405c2b0 0x7ff00405e770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7ff01c060600 tx=0x7ff00c009b30).stop 2021-11-20T06:02:59.372 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.386+0000 7ff021630700 1 -- 172.21.15.139:0/2067117302 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff01c0f67b0 msgr2=0x7ff01c10c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:02:59.372 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.386+0000 7ff021630700 1 --2- 172.21.15.139:0/2067117302 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff01c0f67b0 0x7ff01c10c0c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff0100026d0 tx=0x7ff010009de0).stop 2021-11-20T06:02:59.373 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.386+0000 7ff021630700 1 -- 172.21.15.139:0/2067117302 shutdown_connections 2021-11-20T06:02:59.373 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.386+0000 7ff021630700 1 --2- 172.21.15.139:0/2067117302 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff00405c2b0 0x7ff00405e770 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:59.373 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.386+0000 7ff021630700 1 --2- 172.21.15.139:0/2067117302 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff01c0f5510 0x7ff01c109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:59.373 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.386+0000 7ff021630700 1 --2- 172.21.15.139:0/2067117302 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff01c0f67b0 0x7ff01c10c0c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:02:59.373 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.386+0000 7ff021630700 1 -- 172.21.15.139:0/2067117302 >> 172.21.15.139:0/2067117302 conn(0x7ff01c0f0c70 msgr2=0x7ff01c0fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:02:59.374 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.386+0000 7ff021630700 1 -- 172.21.15.139:0/2067117302 shutdown_connections 2021-11-20T06:02:59.374 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:02:59.386+0000 7ff021630700 1 -- 172.21.15.139:0/2067117302 wait complete. 2021-11-20T06:02:59.787 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149709 got 111669149711 for osd.3 2021-11-20T06:02:59.787 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.4 2021-11-20T06:03:00.224 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:03:00.417 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:00 smithi139 conmon[33501]: cluster 2021-11-20T06:02:58.203615+0000 mgr.smithi139.dplyxn (mgr.14162) 162 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:00.418 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:00 smithi139 conmon[33501]: audit 2021-11-20T06:02:59.384959+0000 mon.smithi139 (mon.0) 440 : audit [DBG] from='client.? 172.21.15.139:0/2067117302' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-11-20T06:03:00.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:00 smithi178 conmon[36600]: cluster 2021-11-20T06:02:58.203615+0000 mgr.smithi139.dplyxn (mgr.14162) 162 2021-11-20T06:03:00.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:00 smithi178 conmon[36600]: : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:00.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:00 smithi178 conmon[36600]: audit 2021-11-20T06:02:59.384959+0000 mon.smithi139 (mon.0) 440 : audit [DBG] from='client.? 172.21.15.139:0/2067117302' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-11-20T06:03:01.370 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.384+0000 7fa518552700 1 -- 172.21.15.139:0/168338458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5100f67d0 msgr2=0x7fa5100f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:01.371 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.384+0000 7fa518552700 1 --2- 172.21.15.139:0/168338458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5100f67d0 0x7fa5100f6bf0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fa50c004660 tx=0x7fa50c009b30).stop 2021-11-20T06:03:01.371 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.384+0000 7fa518552700 1 -- 172.21.15.139:0/168338458 shutdown_connections 2021-11-20T06:03:01.371 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.384+0000 7fa518552700 1 --2- 172.21.15.139:0/168338458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5100f7a70 0x7fa5100f7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:03:01.371 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.384+0000 7fa518552700 1 --2- 172.21.15.139:0/168338458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5100f67d0 0x7fa5100f6bf0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:01.372 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.384+0000 7fa518552700 1 -- 172.21.15.139:0/168338458 >> 172.21.15.139:0/168338458 conn(0x7fa5100f1f50 msgr2=0x7fa5100f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:01.372 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.384+0000 7fa518552700 1 -- 172.21.15.139:0/168338458 shutdown_connections 2021-11-20T06:03:01.372 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.384+0000 7fa518552700 1 -- 172.21.15.139:0/168338458 wait complete. 2021-11-20T06:03:01.373 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:01 smithi139 conmon[33501]: cluster 2021-11-20T06:03:00.204058+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:03:01.373 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:01 smithi139 conmon[33501]: 163 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:01.375 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.385+0000 7fa518552700 1 Processor -- start 2021-11-20T06:03:01.375 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.385+0000 7fa518552700 1 -- start start 2021-11-20T06:03:01.375 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.385+0000 7fa518552700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5100f67d0 0x7fa51006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:01.375 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.385+0000 7fa518552700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5100f7a70 0x7fa510069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:01.376 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.385+0000 7fa518552700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51006a410 con 0x7fa5100f67d0 2021-11-20T06:03:01.376 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.385+0000 7fa518552700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51006a550 con 0x7fa5100f7a70 2021-11-20T06:03:01.376 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.385+0000 7fa5162ee700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5100f67d0 0x7fa51006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:01.376 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.385+0000 7fa5162ee700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5100f67d0 0x7fa51006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57272/0 (socket says 172.21.15.139:57272) 2021-11-20T06:03:01.377 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.385+0000 7fa5162ee700 1 -- 172.21.15.139:0/2612264742 learned_addr learned my addr 172.21.15.139:0/2612264742 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:01.377 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.385+0000 7fa515aed700 1 --2- 172.21.15.139:0/2612264742 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5100f7a70 0x7fa510069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:01.377 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.385+0000 7fa5162ee700 1 -- 172.21.15.139:0/2612264742 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5100f7a70 msgr2=0x7fa510069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:01.377 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.385+0000 7fa5162ee700 1 --2- 172.21.15.139:0/2612264742 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5100f7a70 0x7fa510069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:01.377 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.385+0000 7fa5162ee700 1 -- 172.21.15.139:0/2612264742 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50c005040 con 0x7fa5100f67d0 2021-11-20T06:03:01.378 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.385+0000 7fa5162ee700 1 --2- 172.21.15.139:0/2612264742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5100f67d0 0x7fa51006b9a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa50c009f50 tx=0x7fa50c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:01.378 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.386+0000 7fa506ffd700 1 -- 172.21.15.139:0/2612264742 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa50c018070 con 0x7fa5100f67d0 2021-11-20T06:03:01.378 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.386+0000 7fa518552700 1 -- 172.21.15.139:0/2612264742 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51006a7d0 con 0x7fa5100f67d0 2021-11-20T06:03:01.378 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.386+0000 7fa506ffd700 1 -- 172.21.15.139:0/2612264742 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa50c013410 con 0x7fa5100f67d0 2021-11-20T06:03:01.379 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.386+0000 7fa506ffd700 1 -- 172.21.15.139:0/2612264742 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa50c01c620 con 0x7fa5100f67d0 2021-11-20T06:03:01.379 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.386+0000 7fa518552700 1 -- 172.21.15.139:0/2612264742 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51010f110 con 0x7fa5100f67d0 2021-11-20T06:03:01.379 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.386+0000 7fa518552700 1 -- 172.21.15.139:0/2612264742 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa510045ad0 con 0x7fa5100f67d0 2021-11-20T06:03:01.379 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.387+0000 7fa506ffd700 1 -- 172.21.15.139:0/2612264742 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa50c01d420 con 0x7fa5100f67d0 2021-11-20T06:03:01.379 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.387+0000 7fa506ffd700 1 --2- 172.21.15.139:0/2612264742 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa4fc05c2a0 0x7fa4fc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:01.380 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.387+0000 7fa506ffd700 1 -- 172.21.15.139:0/2612264742 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fa50c077a40 con 0x7fa5100f67d0 2021-11-20T06:03:01.380 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.387+0000 7fa515aed700 1 --2- 172.21.15.139:0/2612264742 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa4fc05c2a0 0x7fa4fc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:01.380 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.388+0000 7fa515aed700 1 --2- 172.21.15.139:0/2612264742 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa4fc05c2a0 0x7fa4fc05e760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fa50000b830 tx=0x7fa500008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:01.381 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.391+0000 7fa506ffd700 1 -- 172.21.15.139:0/2612264742 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa50c04aa40 con 0x7fa5100f67d0 2021-11-20T06:03:01.525 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.539+0000 7fa518552700 1 -- 172.21.15.139:0/2612264742 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa510027a00 con 0x7fa5100f67d0 2021-11-20T06:03:01.525 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.540+0000 7fa506ffd700 1 -- 172.21.15.139:0/2612264742 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa50c04e060 con 0x7fa5100f67d0 2021-11-20T06:03:01.526 INFO:teuthology.orchestra.run.smithi139.stdout:133143986189 2021-11-20T06:03:01.527 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.541+0000 7fa518552700 1 -- 172.21.15.139:0/2612264742 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa4fc05c2a0 msgr2=0x7fa4fc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:01.527 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.541+0000 7fa518552700 1 --2- 172.21.15.139:0/2612264742 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa4fc05c2a0 0x7fa4fc05e760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fa50000b830 tx=0x7fa500008040).stop 2021-11-20T06:03:01.527 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.542+0000 7fa518552700 1 -- 172.21.15.139:0/2612264742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5100f67d0 msgr2=0x7fa51006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:01.528 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.542+0000 7fa518552700 1 --2- 172.21.15.139:0/2612264742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5100f67d0 0x7fa51006b9a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa50c009f50 tx=0x7fa50c009b30).stop 2021-11-20T06:03:01.528 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.542+0000 7fa518552700 1 -- 172.21.15.139:0/2612264742 shutdown_connections 2021-11-20T06:03:01.528 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.542+0000 7fa518552700 1 --2- 172.21.15.139:0/2612264742 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa4fc05c2a0 0x7fa4fc05e760 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:01.528 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.542+0000 7fa518552700 1 --2- 172.21.15.139:0/2612264742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa5100f67d0 0x7fa51006b9a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:01.529 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.542+0000 7fa518552700 1 --2- 172.21.15.139:0/2612264742 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5100f7a70 0x7fa510069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:01.529 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.542+0000 7fa518552700 1 -- 172.21.15.139:0/2612264742 >> 172.21.15.139:0/2612264742 conn(0x7fa5100f1f50 msgr2=0x7fa5100fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:01.529 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.542+0000 7fa518552700 1 -- 172.21.15.139:0/2612264742 shutdown_connections 2021-11-20T06:03:01.529 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:01.542+0000 7fa518552700 1 -- 172.21.15.139:0/2612264742 wait complete. 2021-11-20T06:03:01.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:01 smithi178 conmon[36600]: cluster 2021-11-20T06:03:00.204058+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:03:01.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:01 smithi178 conmon[36600]: ) 163 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:01.833 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986187 got 133143986189 for osd.4 2021-11-20T06:03:01.834 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.5 2021-11-20T06:03:02.272 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:03:02.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:02 smithi178 conmon[36600]: audit 2021-11-20T06:03:01.540908+0000 mon.smithi139 (mon.0) 2021-11-20T06:03:02.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:02 smithi178 conmon[36600]: 441 : audit [DBG] from='client.? 172.21.15.139:0/2612264742' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-11-20T06:03:02.467 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:02 smithi139 conmon[33501]: audit 2021-11-20T06:03:01.540908+0000 mon.smithi139 (mon.0) 441 : audit [DBG] from='client.? 172.21.15.139:0/2612264742' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-11-20T06:03:03.519 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.531+0000 7f14c1904700 1 -- 172.21.15.139:0/1797171808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14bc0f7900 msgr2=0x7f14bc0f7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:03.519 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.531+0000 7f14c1904700 1 --2- 172.21.15.139:0/1797171808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14bc0f7900 0x7f14bc0f7d60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f14a4004660 tx=0x7f14a4009b30).stop 2021-11-20T06:03:03.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.531+0000 7f14c1904700 1 -- 172.21.15.139:0/1797171808 shutdown_connections 2021-11-20T06:03:03.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.531+0000 7f14c1904700 1 --2- 172.21.15.139:0/1797171808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14bc0f7900 0x7f14bc0f7d60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:03.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.531+0000 7f14c1904700 1 --2- 172.21.15.139:0/1797171808 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14bc0f6660 0x7f14bc0f6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:03.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.531+0000 7f14c1904700 1 -- 172.21.15.139:0/1797171808 >> 172.21.15.139:0/1797171808 conn(0x7f14bc0f1de0 msgr2=0x7f14bc0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:03.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.531+0000 7f14c1904700 1 -- 172.21.15.139:0/1797171808 shutdown_connections 2021-11-20T06:03:03.521 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.531+0000 7f14c1904700 1 -- 172.21.15.139:0/1797171808 wait complete. 2021-11-20T06:03:03.521 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.531+0000 7f14c1904700 1 Processor -- start 2021-11-20T06:03:03.521 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.531+0000 7f14c1904700 1 -- start start 2021-11-20T06:03:03.521 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.531+0000 7f14c1904700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14bc0f6660 0x7f14bc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:03.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.531+0000 7f14c1904700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14bc0f7900 0x7f14bc069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:03.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.531+0000 7f14c1904700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc06a380 con 0x7f14bc0f7900 2021-11-20T06:03:03.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.531+0000 7f14c1904700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc06a4c0 con 0x7f14bc0f6660 2021-11-20T06:03:03.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.532+0000 7f14b3fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14bc0f7900 0x7f14bc069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:03.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.532+0000 7f14b3fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14bc0f7900 0x7f14bc069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57284/0 (socket says 172.21.15.139:57284) 2021-11-20T06:03:03.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.532+0000 7f14b3fff700 1 -- 172.21.15.139:0/3103534536 learned_addr learned my addr 172.21.15.139:0/3103534536 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:03.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.532+0000 7f14baffd700 1 --2- 172.21.15.139:0/3103534536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14bc0f6660 0x7f14bc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:03.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.532+0000 7f14b3fff700 1 -- 172.21.15.139:0/3103534536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14bc0f6660 msgr2=0x7f14bc06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:03.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.532+0000 7f14b3fff700 1 --2- 172.21.15.139:0/3103534536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14bc0f6660 0x7f14bc06b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:03.524 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.532+0000 7f14b3fff700 1 -- 172.21.15.139:0/3103534536 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a4005040 con 0x7f14bc0f7900 2021-11-20T06:03:03.524 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.532+0000 7f14baffd700 1 --2- 172.21.15.139:0/3103534536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14bc0f6660 0x7f14bc06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:03:03.524 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.532+0000 7f14b3fff700 1 --2- 172.21.15.139:0/3103534536 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14bc0f7900 0x7f14bc069e20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f14a4009f50 tx=0x7f14a4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:03.524 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.532+0000 7f14b37fe700 1 -- 172.21.15.139:0/3103534536 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14a4018070 con 0x7f14bc0f7900 2021-11-20T06:03:03.524 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.532+0000 7f14b37fe700 1 -- 172.21.15.139:0/3103534536 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14a4013410 con 0x7f14bc0f7900 2021-11-20T06:03:03.525 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.532+0000 7f14c1904700 1 -- 172.21.15.139:0/3103534536 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14bc06a740 con 0x7f14bc0f7900 2021-11-20T06:03:03.525 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.532+0000 7f14b37fe700 1 -- 172.21.15.139:0/3103534536 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14a401c3c0 con 0x7f14bc0f7900 2021-11-20T06:03:03.525 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.532+0000 7f14c1904700 1 -- 172.21.15.139:0/3103534536 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14bc06aca0 con 0x7f14bc0f7900 2021-11-20T06:03:03.525 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.533+0000 7f14c1904700 1 -- 172.21.15.139:0/3103534536 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14bc045ad0 con 0x7f14bc0f7900 2021-11-20T06:03:03.525 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.533+0000 7f14b37fe700 1 -- 172.21.15.139:0/3103534536 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f14a401d420 con 0x7f14bc0f7900 2021-11-20T06:03:03.526 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.534+0000 7f14b37fe700 1 --2- 172.21.15.139:0/3103534536 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f14a805c2b0 0x7f14a805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:03.526 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.534+0000 7f14b37fe700 1 -- 172.21.15.139:0/3103534536 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f14a40777a0 con 0x7f14bc0f7900 2021-11-20T06:03:03.527 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.535+0000 7f14baffd700 1 --2- 172.21.15.139:0/3103534536 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f14a805c2b0 0x7f14a805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:03.527 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.535+0000 7f14baffd700 1 --2- 172.21.15.139:0/3103534536 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f14a805c2b0 0x7f14a805e770 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f14ac009fd0 tx=0x7f14ac00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:03.527 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.538+0000 7f14b37fe700 1 -- 172.21.15.139:0/3103534536 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14a404a820 con 0x7f14bc0f7900 2021-11-20T06:03:03.672 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.687+0000 7f14c1904700 1 -- 172.21.15.139:0/3103534536 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f14bc0fef30 con 0x7f14bc0f7900 2021-11-20T06:03:03.673 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.687+0000 7f14b37fe700 1 -- 172.21.15.139:0/3103534536 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f14a404de40 con 0x7f14bc0f7900 2021-11-20T06:03:03.673 INFO:teuthology.orchestra.run.smithi139.stdout:150323855370 2021-11-20T06:03:03.674 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.689+0000 7f14c1904700 1 -- 172.21.15.139:0/3103534536 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f14a805c2b0 msgr2=0x7f14a805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:03.674 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.689+0000 7f14c1904700 1 --2- 172.21.15.139:0/3103534536 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f14a805c2b0 0x7f14a805e770 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f14ac009fd0 tx=0x7f14ac00b040).stop 2021-11-20T06:03:03.675 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.689+0000 7f14c1904700 1 -- 172.21.15.139:0/3103534536 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14bc0f7900 msgr2=0x7f14bc069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:03.675 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.689+0000 7f14c1904700 1 --2- 172.21.15.139:0/3103534536 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14bc0f7900 0x7f14bc069e20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f14a4009f50 tx=0x7f14a4009b30).stop 2021-11-20T06:03:03.675 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.689+0000 7f14c1904700 1 -- 172.21.15.139:0/3103534536 shutdown_connections 2021-11-20T06:03:03.676 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.689+0000 7f14c1904700 1 --2- 172.21.15.139:0/3103534536 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f14a805c2b0 0x7f14a805e770 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:03.676 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.689+0000 7f14c1904700 1 --2- 172.21.15.139:0/3103534536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14bc0f6660 0x7f14bc06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:03.676 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.689+0000 7f14c1904700 1 --2- 172.21.15.139:0/3103534536 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14bc0f7900 0x7f14bc069e20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:03.677 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.689+0000 7f14c1904700 1 -- 172.21.15.139:0/3103534536 >> 172.21.15.139:0/3103534536 conn(0x7f14bc0f1de0 msgr2=0x7f14bc0fb350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:03.677 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.689+0000 7f14c1904700 1 -- 172.21.15.139:0/3103534536 shutdown_connections 2021-11-20T06:03:03.677 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:03.689+0000 7f14c1904700 1 -- 172.21.15.139:0/3103534536 wait complete. 2021-11-20T06:03:03.982 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855368 got 150323855370 for osd.5 2021-11-20T06:03:03.986 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2021-11-20T06:03:03.988 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph pg dump --format=json 2021-11-20T06:03:04.422 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:03:04.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:04 smithi178 conmon[36600]: cluster 2021-11-20T06:03:02.204416+0000 mgr.smithi139.dplyxn (mgr.14162) 164 : cluster [DBG] 2021-11-20T06:03:04.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:04 smithi178 conmon[36600]: pgmap v133: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:04.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:04 smithi178 conmon[36600]: audit 2021-11-20T06:03:03.688258+0000 mon.smithi139 (mon.0) 442 : audit [DBG] from='client.? 172.21.15.139:0/3103534536' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-11-20T06:03:04.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:04 smithi139 conmon[33501]: cluster 2021-11-20T06:03:02.204416+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:03:04.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:04 smithi139 conmon[33501]: 14162) 164 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:04.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:04 smithi139 conmon[33501]: audit 2021-11-20T06:03:03.688258+0000 mon.smithi139 (mon.0) 442 : audit [DBG] from='client.? 172.21.15.139:0/3103534536' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-11-20T06:03:05.569 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.581+0000 7fee1ce4f700 1 -- 172.21.15.139:0/1962479360 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee180f67e0 msgr2=0x7fee180f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:05.570 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.581+0000 7fee1ce4f700 1 --2- 172.21.15.139:0/1962479360 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee180f67e0 0x7fee180f6c00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fee0c004660 tx=0x7fee0c009b30).stop 2021-11-20T06:03:05.570 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.581+0000 7fee1ce4f700 1 -- 172.21.15.139:0/1962479360 shutdown_connections 2021-11-20T06:03:05.570 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.581+0000 7fee1ce4f700 1 --2- 172.21.15.139:0/1962479360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee180f7a80 0x7fee180f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:05.570 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.581+0000 7fee1ce4f700 1 --2- 172.21.15.139:0/1962479360 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee180f67e0 0x7fee180f6c00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:05.571 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.581+0000 7fee1ce4f700 1 -- 172.21.15.139:0/1962479360 >> 172.21.15.139:0/1962479360 conn(0x7fee180f1f20 msgr2=0x7fee180f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:05.571 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.581+0000 7fee1ce4f700 1 -- 172.21.15.139:0/1962479360 shutdown_connections 2021-11-20T06:03:05.571 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.581+0000 7fee1ce4f700 1 -- 172.21.15.139:0/1962479360 wait complete. 2021-11-20T06:03:05.571 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.581+0000 7fee1ce4f700 1 Processor -- start 2021-11-20T06:03:05.572 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee1ce4f700 1 -- start start 2021-11-20T06:03:05.572 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee1ce4f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee180f67e0 0x7fee181070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:05.573 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee1ce4f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee180f7a80 0x7fee181075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:05.573 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee1ce4f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee18107b70 con 0x7fee180f67e0 2021-11-20T06:03:05.573 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee1ce4f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee18107cb0 con 0x7fee180f7a80 2021-11-20T06:03:05.574 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee1659c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee180f67e0 0x7fee181070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:05.574 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee1659c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee180f67e0 0x7fee181070d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57292/0 (socket says 172.21.15.139:57292) 2021-11-20T06:03:05.575 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee1659c700 1 -- 172.21.15.139:0/86044164 learned_addr learned my addr 172.21.15.139:0/86044164 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:05.575 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee1659c700 1 -- 172.21.15.139:0/86044164 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee180f7a80 msgr2=0x7fee181075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:05.575 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee15d9b700 1 --2- 172.21.15.139:0/86044164 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee180f7a80 0x7fee181075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:05.576 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee1659c700 1 --2- 172.21.15.139:0/86044164 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee180f7a80 0x7fee181075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:05.576 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee1659c700 1 -- 172.21.15.139:0/86044164 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c005040 con 0x7fee180f67e0 2021-11-20T06:03:05.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee1659c700 1 --2- 172.21.15.139:0/86044164 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee180f67e0 0x7fee181070d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fee0c009f50 tx=0x7fee0c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:05.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee06ffd700 1 -- 172.21.15.139:0/86044164 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee0c018070 con 0x7fee180f67e0 2021-11-20T06:03:05.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee06ffd700 1 -- 172.21.15.139:0/86044164 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee0c013410 con 0x7fee180f67e0 2021-11-20T06:03:05.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee1ce4f700 1 -- 172.21.15.139:0/86044164 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee181085f0 con 0x7fee180f67e0 2021-11-20T06:03:05.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee06ffd700 1 -- 172.21.15.139:0/86044164 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee0c01c620 con 0x7fee180f67e0 2021-11-20T06:03:05.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.582+0000 7fee1ce4f700 1 -- 172.21.15.139:0/86044164 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee18108b50 con 0x7fee180f67e0 2021-11-20T06:03:05.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.583+0000 7fee1ce4f700 1 -- 172.21.15.139:0/86044164 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee18045ad0 con 0x7fee180f67e0 2021-11-20T06:03:05.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.584+0000 7fee06ffd700 1 -- 172.21.15.139:0/86044164 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fee0c01d420 con 0x7fee180f67e0 2021-11-20T06:03:05.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.584+0000 7fee06ffd700 1 --2- 172.21.15.139:0/86044164 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fee0005c2a0 0x7fee0005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:05.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.584+0000 7fee06ffd700 1 -- 172.21.15.139:0/86044164 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fee0c0779f0 con 0x7fee180f67e0 2021-11-20T06:03:05.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.584+0000 7fee15d9b700 1 --2- 172.21.15.139:0/86044164 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fee0005c2a0 0x7fee0005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:05.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.585+0000 7fee15d9b700 1 --2- 172.21.15.139:0/86044164 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fee0005c2a0 0x7fee0005e760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fee080041d0 tx=0x7fee08009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:05.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.588+0000 7fee06ffd700 1 -- 172.21.15.139:0/86044164 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee0c079e60 con 0x7fee180f67e0 2021-11-20T06:03:05.719 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.733+0000 7fee1ce4f700 1 -- 172.21.15.139:0/86044164 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fee180ff120 con 0x7fee0005c2a0 2021-11-20T06:03:05.720 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.735+0000 7fee06ffd700 1 -- 172.21.15.139:0/86044164 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24057 (secure 0 0 0) 0x7fee080044e0 con 0x7fee0005c2a0 2021-11-20T06:03:05.721 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:03:05.722 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.736+0000 7fee1ce4f700 1 -- 172.21.15.139:0/86044164 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fee0005c2a0 msgr2=0x7fee0005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:05.723 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.736+0000 7fee1ce4f700 1 --2- 172.21.15.139:0/86044164 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fee0005c2a0 0x7fee0005e760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fee080041d0 tx=0x7fee08009040).stop 2021-11-20T06:03:05.723 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.736+0000 7fee1ce4f700 1 -- 172.21.15.139:0/86044164 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee180f67e0 msgr2=0x7fee181070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:05.723 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.736+0000 7fee1ce4f700 1 --2- 172.21.15.139:0/86044164 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee180f67e0 0x7fee181070d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fee0c009f50 tx=0x7fee0c009b30).stop 2021-11-20T06:03:05.724 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.736+0000 7fee1ce4f700 1 -- 172.21.15.139:0/86044164 shutdown_connections 2021-11-20T06:03:05.724 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.736+0000 7fee1ce4f700 1 --2- 172.21.15.139:0/86044164 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fee0005c2a0 0x7fee0005e760 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:05.726 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.736+0000 7fee1ce4f700 1 --2- 172.21.15.139:0/86044164 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fee180f67e0 0x7fee181070d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:05.726 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.736+0000 7fee1ce4f700 1 --2- 172.21.15.139:0/86044164 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee180f7a80 0x7fee181075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:05.726 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.736+0000 7fee1ce4f700 1 -- 172.21.15.139:0/86044164 >> 172.21.15.139:0/86044164 conn(0x7fee180f1f20 msgr2=0x7fee180fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:05.727 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.736+0000 7fee1ce4f700 1 -- 172.21.15.139:0/86044164 shutdown_connections 2021-11-20T06:03:05.727 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:05.737+0000 7fee1ce4f700 1 -- 172.21.15.139:0/86044164 wait complete. 2021-11-20T06:03:05.727 INFO:teuthology.orchestra.run.smithi139.stderr:dumped all 2021-11-20T06:03:06.223 INFO:teuthology.orchestra.run.smithi139.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2021-11-20T06:03:04.204572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":30936,"kb_used_data":1896,"kb_used_omap":0,"kb_used_meta":28992,"kb_avail":562439976,"statfs":{"total":575970213888,"available":575938535424,"internally_reserved":0,"allocated":1941504,"data_stored":722634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29687808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"34","reported_epoch":"35","state":"active+clean","last_fresh":"2021-11-20T06:02:27.185035+0000","last_change":"2021-11-20T06:02:10.210693+0000","last_active":"2021-11-20T06:02:27.185035+0000","last_peered":"2021-11-20T06:02:27.185035+0000","last_clean":"2021-11-20T06:02:27.185035+0000","last_became_active":"2021-11-20T06:02:10.210425+0000","last_became_peered":"2021-11-20T06:02:10.210425+0000","last_unstale":"2021-11-20T06:02:27.185035+0000","last_undegraded":"2021-11-20T06:02:27.185035+0000","last_fullsized":"2021-11-20T06:02:27.185035+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-11-20T06:01:49.138321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-11-20T06:01:49.138321+0000","last_clean_scrub_stamp":"2021-11-20T06:01:49.138321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":35,"seq":150323855371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5060,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740092,"statfs":{"total":95995035648,"available":95989854208,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5060,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740092,"statfs":{"total":95995035648,"available":95989854208,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]}]},{"osd":3,"up_from":26,"seq":111669149712,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5124,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740028,"statfs":{"total":95995035648,"available":95989788672,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 06:02:54 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.60699999999999998}]},{"osd":1,"last update":"Sat Nov 20 06:02:54 2021","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.56200000000000006}]},{"osd":2,"last update":"Sat Nov 20 06:02:54 2021","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.75900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411347,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 06:02:40 2021","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.441}]},{"osd":1,"last update":"Sat Nov 20 06:02:40 2021","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.48699999999999999}]},{"osd":3,"last update":"Sat Nov 20 06:02:54 2021","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.58399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705691,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5252,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739900,"statfs":{"total":95995035648,"available":95989657600,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Nov 20 06:02:23 2021","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.30099999999999999}]},{"osd":2,"last update":"Sat Nov 20 06:02:43 2021","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.28999999999999998}]},{"osd":3,"last update":"Sat Nov 20 06:02:56 2021","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.43099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542168,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5252,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739900,"statfs":{"total":95995035648,"available":95989657600,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 06:02:22 2021","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.434,"5min":0.434,"15min":0.434},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"last":0.27100000000000002}]},{"osd":2,"last update":"Sat Nov 20 06:02:42 2021","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.315,"15min":0.315},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.316,"5min":0.316,"15min":0.316},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.307}]},{"osd":3,"last update":"Sat Nov 20 06:02:56 2021","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.45200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-11-20T06:03:06.225 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph pg dump --format=json 2021-11-20T06:03:06.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:06 smithi178 conmon[36600]: cluster 2021-11-20T06:03:04.204767+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:03:06.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:06 smithi178 conmon[36600]: ) 165 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:06.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:06 smithi139 conmon[33501]: cluster 2021-11-20T06:03:04.204767+0000 mgr.smithi139.dplyxn (mgr.14162) 165 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:06.659 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:03:07.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:07 smithi178 conmon[36600]: audit 2021-11-20T06:03:05.734838+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:03:07.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:07 smithi178 conmon[36600]: ) 166 : audit [DBG] from='client.14378 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T06:03:07.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:07 smithi178 conmon[36600]: cluster 2021-11-20T06:03:06.205089+0000 mgr.smithi139.dplyxn (mgr.14162) 167 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:07.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:07 smithi139 conmon[33501]: audit 2021-11-20T06:03:05.734838+0000 mgr.smithi139.dplyxn (mgr.14162) 166 : audit [DBG] from='client.14378 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T06:03:07.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:07 smithi139 conmon[33501]: cluster 2021-11-20T06:03:06.205089+0000 mgr.smithi139.dplyxn (mgr.14162) 167 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:07.918 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.929+0000 7f32d23af700 1 -- 172.21.15.139:0/25584291 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32cc0f56e0 msgr2=0x7f32cc0f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:07.918 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.929+0000 7f32d23af700 1 --2- 172.21.15.139:0/25584291 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32cc0f56e0 0x7f32cc0f5b00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f32bc004660 tx=0x7f32bc009b30).stop 2021-11-20T06:03:07.919 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.929+0000 7f32d23af700 1 -- 172.21.15.139:0/25584291 shutdown_connections 2021-11-20T06:03:07.919 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.929+0000 7f32d23af700 1 --2- 172.21.15.139:0/25584291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32cc0f6980 0x7f32cc0f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:07.919 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.929+0000 7f32d23af700 1 --2- 172.21.15.139:0/25584291 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32cc0f56e0 0x7f32cc0f5b00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:07.919 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.929+0000 7f32d23af700 1 -- 172.21.15.139:0/25584291 >> 172.21.15.139:0/25584291 conn(0x7f32cc0f0e20 msgr2=0x7f32cc0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:07.920 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.929+0000 7f32d23af700 1 -- 172.21.15.139:0/25584291 shutdown_connections 2021-11-20T06:03:07.920 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.929+0000 7f32d23af700 1 -- 172.21.15.139:0/25584291 wait complete. 2021-11-20T06:03:07.920 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.930+0000 7f32d23af700 1 Processor -- start 2021-11-20T06:03:07.920 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.930+0000 7f32d23af700 1 -- start start 2021-11-20T06:03:07.920 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.930+0000 7f32d23af700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32cc0f56e0 0x7f32cc10d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:07.921 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.930+0000 7f32d23af700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32cc0f6980 0x7f32cc10dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:07.921 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.930+0000 7f32d23af700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc10e170 con 0x7f32cc0f6980 2021-11-20T06:03:07.921 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.930+0000 7f32d23af700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc10e2b0 con 0x7f32cc0f56e0 2021-11-20T06:03:07.921 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.931+0000 7f32cb7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32cc0f6980 0x7f32cc10dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:07.922 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.931+0000 7f32cb7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32cc0f6980 0x7f32cc10dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57302/0 (socket says 172.21.15.139:57302) 2021-11-20T06:03:07.922 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.931+0000 7f32cb7fe700 1 -- 172.21.15.139:0/1430567459 learned_addr learned my addr 172.21.15.139:0/1430567459 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:07.922 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.931+0000 7f32cbfff700 1 --2- 172.21.15.139:0/1430567459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32cc0f56e0 0x7f32cc10d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:07.922 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.931+0000 7f32cb7fe700 1 -- 172.21.15.139:0/1430567459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32cc0f56e0 msgr2=0x7f32cc10d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:07.922 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.931+0000 7f32cb7fe700 1 --2- 172.21.15.139:0/1430567459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32cc0f56e0 0x7f32cc10d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:07.923 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.931+0000 7f32cb7fe700 1 -- 172.21.15.139:0/1430567459 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32bc005040 con 0x7f32cc0f6980 2021-11-20T06:03:07.923 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.931+0000 7f32cb7fe700 1 --2- 172.21.15.139:0/1430567459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32cc0f6980 0x7f32cc10dbc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f32c000e7f0 tx=0x7f32c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:07.923 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.931+0000 7f32c8ff9700 1 -- 172.21.15.139:0/1430567459 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32c00092e0 con 0x7f32cc0f6980 2021-11-20T06:03:07.923 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.931+0000 7f32c8ff9700 1 -- 172.21.15.139:0/1430567459 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32c000f040 con 0x7f32cc0f6980 2021-11-20T06:03:07.923 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.931+0000 7f32d23af700 1 -- 172.21.15.139:0/1430567459 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32cc10ec50 con 0x7f32cc0f6980 2021-11-20T06:03:07.924 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.931+0000 7f32c8ff9700 1 -- 172.21.15.139:0/1430567459 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32c0014820 con 0x7f32cc0f6980 2021-11-20T06:03:07.924 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.931+0000 7f32d23af700 1 -- 172.21.15.139:0/1430567459 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32cc10f210 con 0x7f32cc0f6980 2021-11-20T06:03:07.925 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.932+0000 7f32d23af700 1 -- 172.21.15.139:0/1430567459 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32cc045ad0 con 0x7f32cc0f6980 2021-11-20T06:03:07.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.933+0000 7f32c8ff9700 1 -- 172.21.15.139:0/1430567459 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f32c0014a90 con 0x7f32cc0f6980 2021-11-20T06:03:07.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.933+0000 7f32c8ff9700 1 --2- 172.21.15.139:0/1430567459 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f32b405c2a0 0x7f32b405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:07.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.933+0000 7f32c8ff9700 1 -- 172.21.15.139:0/1430567459 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f32c0075b90 con 0x7f32cc0f6980 2021-11-20T06:03:07.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.933+0000 7f32cbfff700 1 --2- 172.21.15.139:0/1430567459 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f32b405c2a0 0x7f32b405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:07.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.933+0000 7f32cbfff700 1 --2- 172.21.15.139:0/1430567459 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f32b405c2a0 0x7f32b405e760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f32bc013530 tx=0x7f32bc016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:07.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:07.937+0000 7f32c8ff9700 1 -- 172.21.15.139:0/1430567459 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32c00472b0 con 0x7f32cc0f6980 2021-11-20T06:03:08.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:08.081+0000 7f32d23af700 1 -- 172.21.15.139:0/1430567459 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f32cc0fb3e0 con 0x7f32b405c2a0 2021-11-20T06:03:08.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:08.082+0000 7f32c8ff9700 1 -- 172.21.15.139:0/1430567459 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24071 (secure 0 0 0) 0x7f32cc0fb3e0 con 0x7f32b405c2a0 2021-11-20T06:03:08.070 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:03:08.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:08.084+0000 7f32d23af700 1 -- 172.21.15.139:0/1430567459 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f32b405c2a0 msgr2=0x7f32b405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:08.072 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:08.084+0000 7f32d23af700 1 --2- 172.21.15.139:0/1430567459 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f32b405c2a0 0x7f32b405e760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f32bc013530 tx=0x7f32bc016040).stop 2021-11-20T06:03:08.072 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:08.084+0000 7f32d23af700 1 -- 172.21.15.139:0/1430567459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32cc0f6980 msgr2=0x7f32cc10dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:08.072 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:08.084+0000 7f32d23af700 1 --2- 172.21.15.139:0/1430567459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32cc0f6980 0x7f32cc10dbc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f32c000e7f0 tx=0x7f32c000b040).stop 2021-11-20T06:03:08.072 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:08.084+0000 7f32d23af700 1 -- 172.21.15.139:0/1430567459 shutdown_connections 2021-11-20T06:03:08.073 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:08.084+0000 7f32d23af700 1 --2- 172.21.15.139:0/1430567459 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f32b405c2a0 0x7f32b405e760 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:08.073 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:08.084+0000 7f32d23af700 1 --2- 172.21.15.139:0/1430567459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32cc0f56e0 0x7f32cc10d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:08.073 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:08.084+0000 7f32d23af700 1 --2- 172.21.15.139:0/1430567459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f32cc0f6980 0x7f32cc10dbc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:08.073 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:08.084+0000 7f32d23af700 1 -- 172.21.15.139:0/1430567459 >> 172.21.15.139:0/1430567459 conn(0x7f32cc0f0e20 msgr2=0x7f32cc05fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:08.074 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:08.084+0000 7f32d23af700 1 -- 172.21.15.139:0/1430567459 shutdown_connections 2021-11-20T06:03:08.074 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:08.084+0000 7f32d23af700 1 -- 172.21.15.139:0/1430567459 wait complete. 2021-11-20T06:03:08.074 INFO:teuthology.orchestra.run.smithi139.stderr:dumped all 2021-11-20T06:03:08.428 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:08 smithi139 conmon[33501]: audit 2021-11-20T06:03:08.083038+0000 mgr.smithi139.dplyxn 2021-11-20T06:03:08.428 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:08 smithi139 conmon[33501]: (mgr.14162) 168 : audit [DBG] from='client.14382 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T06:03:08.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:08 smithi178 conmon[36600]: audit 2021-11-20T06:03:08.083038+0000 mgr.smithi139.dplyxn (mgr.14162) 168 2021-11-20T06:03:08.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:08 smithi178 conmon[36600]: : audit [DBG] from='client.14382 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-20T06:03:08.469 INFO:teuthology.orchestra.run.smithi139.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2021-11-20T06:03:06.204951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":30936,"kb_used_data":1896,"kb_used_omap":0,"kb_used_meta":28992,"kb_avail":562439976,"statfs":{"total":575970213888,"available":575938535424,"internally_reserved":0,"allocated":1941504,"data_stored":722634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29687808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"34","reported_epoch":"35","state":"active+clean","last_fresh":"2021-11-20T06:02:27.185035+0000","last_change":"2021-11-20T06:02:10.210693+0000","last_active":"2021-11-20T06:02:27.185035+0000","last_peered":"2021-11-20T06:02:27.185035+0000","last_clean":"2021-11-20T06:02:27.185035+0000","last_became_active":"2021-11-20T06:02:10.210425+0000","last_became_peered":"2021-11-20T06:02:10.210425+0000","last_unstale":"2021-11-20T06:02:27.185035+0000","last_undegraded":"2021-11-20T06:02:27.185035+0000","last_fullsized":"2021-11-20T06:02:27.185035+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-11-20T06:01:49.138321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-11-20T06:01:49.138321+0000","last_clean_scrub_stamp":"2021-11-20T06:01:49.138321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":35,"seq":150323855371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5060,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740092,"statfs":{"total":95995035648,"available":95989854208,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986191,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5060,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740092,"statfs":{"total":95995035648,"available":95989854208,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5124,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740028,"statfs":{"total":95995035648,"available":95989788672,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 06:02:54 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.45300000000000001}]},{"osd":1,"last update":"Sat Nov 20 06:02:54 2021","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.434},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.46999999999999997}]},{"osd":2,"last update":"Sat Nov 20 06:02:54 2021","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.497},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.56499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411348,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 06:02:40 2021","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.33500000000000002}]},{"osd":1,"last update":"Sat Nov 20 06:02:40 2021","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.309,"15min":0.309},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.35899999999999999}]},{"osd":3,"last update":"Sat Nov 20 06:02:54 2021","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.51300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705691,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5252,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739900,"statfs":{"total":95995035648,"available":95989657600,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Nov 20 06:02:23 2021","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.30099999999999999}]},{"osd":2,"last update":"Sat Nov 20 06:02:43 2021","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.28999999999999998}]},{"osd":3,"last update":"Sat Nov 20 06:02:56 2021","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.43099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542168,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5252,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739900,"statfs":{"total":95995035648,"available":95989657600,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Nov 20 06:02:22 2021","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.434,"5min":0.434,"15min":0.434},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"last":0.27100000000000002}]},{"osd":2,"last update":"Sat Nov 20 06:02:42 2021","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.315,"15min":0.315},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.316,"5min":0.316,"15min":0.316},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.307}]},{"osd":3,"last update":"Sat Nov 20 06:02:56 2021","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.45200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-11-20T06:03:08.470 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2021-11-20T06:03:08.471 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-11-20T06:03:08.471 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2021-11-20T06:03:08.471 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- ceph health --format=json 2021-11-20T06:03:08.905 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/mon.smithi139/config 2021-11-20T06:03:09.297 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:09 smithi139 conmon[33501]: cluster 2021-11-20T06:03:08.205481+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:03:09.298 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:09 smithi139 conmon[33501]: 169 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:09.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:09 smithi178 conmon[36600]: cluster 2021-11-20T06:03:08.205481+0000 mgr.smithi139.dplyxn (mgr.14162) 169 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:10.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.246+0000 7f0e9cc87700 1 -- 172.21.15.139:0/3069506665 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e980f6d60 msgr2=0x7f0e980f7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:10.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.246+0000 7f0e9cc87700 1 --2- 172.21.15.139:0/3069506665 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e980f6d60 0x7f0e980f7180 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f0e88004660 tx=0x7f0e88009b30).stop 2021-11-20T06:03:10.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.247+0000 7f0e9cc87700 1 -- 172.21.15.139:0/3069506665 shutdown_connections 2021-11-20T06:03:10.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.247+0000 7f0e9cc87700 1 --2- 172.21.15.139:0/3069506665 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e980f5a70 0x7f0e980f5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:03:10.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.247+0000 7f0e9cc87700 1 --2- 172.21.15.139:0/3069506665 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e980f6d60 0x7f0e980f7180 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:10.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.247+0000 7f0e9cc87700 1 -- 172.21.15.139:0/3069506665 >> 172.21.15.139:0/3069506665 conn(0x7f0e980f05e0 msgr2=0x7f0e980f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:10.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.247+0000 7f0e9cc87700 1 -- 172.21.15.139:0/3069506665 shutdown_connections 2021-11-20T06:03:10.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.247+0000 7f0e9cc87700 1 -- 172.21.15.139:0/3069506665 wait complete. 2021-11-20T06:03:10.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.247+0000 7f0e9cc87700 1 Processor -- start 2021-11-20T06:03:10.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.247+0000 7f0e9cc87700 1 -- start start 2021-11-20T06:03:10.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.247+0000 7f0e9cc87700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e980f5a70 0x7f0e981070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:10.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.247+0000 7f0e9cc87700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e980f6d60 0x7f0e981075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:10.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.247+0000 7f0e9cc87700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e98107c00 con 0x7f0e980f6d60 2021-11-20T06:03:10.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.247+0000 7f0e9cc87700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e981084b0 con 0x7f0e980f5a70 2021-11-20T06:03:10.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.247+0000 7f0e95d9b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e980f6d60 0x7f0e981075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:10.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.247+0000 7f0e95d9b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e980f6d60 0x7f0e981075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57312/0 (socket says 172.21.15.139:57312) 2021-11-20T06:03:10.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.247+0000 7f0e95d9b700 1 -- 172.21.15.139:0/1893218121 learned_addr learned my addr 172.21.15.139:0/1893218121 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:10.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.248+0000 7f0e9659c700 1 --2- 172.21.15.139:0/1893218121 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e980f5a70 0x7f0e981070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:10.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.248+0000 7f0e95d9b700 1 -- 172.21.15.139:0/1893218121 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e980f5a70 msgr2=0x7f0e981070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:10.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.248+0000 7f0e95d9b700 1 --2- 172.21.15.139:0/1893218121 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e980f5a70 0x7f0e981070d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:10.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.248+0000 7f0e95d9b700 1 -- 172.21.15.139:0/1893218121 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e88005040 con 0x7f0e980f6d60 2021-11-20T06:03:10.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.248+0000 7f0e9659c700 1 --2- 172.21.15.139:0/1893218121 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e980f5a70 0x7f0e981070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:03:10.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.248+0000 7f0e95d9b700 1 --2- 172.21.15.139:0/1893218121 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e980f6d60 0x7f0e981075c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f0e8000b410 tx=0x7f0e80006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:10.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.248+0000 7f0e8effd700 1 -- 172.21.15.139:0/1893218121 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e800125f0 con 0x7f0e980f6d60 2021-11-20T06:03:10.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.248+0000 7f0e9cc87700 1 -- 172.21.15.139:0/1893218121 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e98108790 con 0x7f0e980f6d60 2021-11-20T06:03:10.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.248+0000 7f0e8effd700 1 -- 172.21.15.139:0/1893218121 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e80012c00 con 0x7f0e980f6d60 2021-11-20T06:03:10.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.248+0000 7f0e8effd700 1 -- 172.21.15.139:0/1893218121 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e80002aa0 con 0x7f0e980f6d60 2021-11-20T06:03:10.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.248+0000 7f0e9cc87700 1 -- 172.21.15.139:0/1893218121 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e98108d50 con 0x7f0e980f6d60 2021-11-20T06:03:10.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.249+0000 7f0e9cc87700 1 -- 172.21.15.139:0/1893218121 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e98059100 con 0x7f0e980f6d60 2021-11-20T06:03:10.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.249+0000 7f0e8effd700 1 -- 172.21.15.139:0/1893218121 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0e8000f030 con 0x7f0e980f6d60 2021-11-20T06:03:10.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.249+0000 7f0e8effd700 1 --2- 172.21.15.139:0/1893218121 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0e8405c2a0 0x7f0e8405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:10.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.250+0000 7f0e8effd700 1 -- 172.21.15.139:0/1893218121 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f0e80076060 con 0x7f0e980f6d60 2021-11-20T06:03:10.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.250+0000 7f0e9659c700 1 --2- 172.21.15.139:0/1893218121 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0e8405c2a0 0x7f0e8405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:10.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.250+0000 7f0e9659c700 1 --2- 172.21.15.139:0/1893218121 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0e8405c2a0 0x7f0e8405e760 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f0e88009d60 tx=0x7f0e88009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:10.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.253+0000 7f0e8effd700 1 -- 172.21.15.139:0/1893218121 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e80049000 con 0x7f0e980f6d60 2021-11-20T06:03:10.447 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.460+0000 7f0e9cc87700 1 -- 172.21.15.139:0/1893218121 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0e9810d6f0 con 0x7f0e980f6d60 2021-11-20T06:03:10.447 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.460+0000 7f0e8effd700 1 -- 172.21.15.139:0/1893218121 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f0e8004c620 con 0x7f0e980f6d60 2021-11-20T06:03:10.447 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:03:10.449 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.462+0000 7f0e9cc87700 1 -- 172.21.15.139:0/1893218121 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0e8405c2a0 msgr2=0x7f0e8405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:10.449 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.462+0000 7f0e9cc87700 1 --2- 172.21.15.139:0/1893218121 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0e8405c2a0 0x7f0e8405e760 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f0e88009d60 tx=0x7f0e88009b30).stop 2021-11-20T06:03:10.449 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.462+0000 7f0e9cc87700 1 -- 172.21.15.139:0/1893218121 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e980f6d60 msgr2=0x7f0e981075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:10.449 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.462+0000 7f0e9cc87700 1 --2- 172.21.15.139:0/1893218121 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e980f6d60 0x7f0e981075c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f0e8000b410 tx=0x7f0e80006ef0).stop 2021-11-20T06:03:10.450 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.462+0000 7f0e9cc87700 1 -- 172.21.15.139:0/1893218121 shutdown_connections 2021-11-20T06:03:10.450 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.462+0000 7f0e9cc87700 1 --2- 172.21.15.139:0/1893218121 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0e8405c2a0 0x7f0e8405e760 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:10.450 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.462+0000 7f0e9cc87700 1 --2- 172.21.15.139:0/1893218121 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e980f5a70 0x7f0e981070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:10.450 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.462+0000 7f0e9cc87700 1 --2- 172.21.15.139:0/1893218121 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0e980f6d60 0x7f0e981075c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:10.451 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.462+0000 7f0e9cc87700 1 -- 172.21.15.139:0/1893218121 >> 172.21.15.139:0/1893218121 conn(0x7f0e980f05e0 msgr2=0x7f0e980f5140 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:10.451 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.463+0000 7f0e9cc87700 1 -- 172.21.15.139:0/1893218121 shutdown_connections 2021-11-20T06:03:10.451 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:10.463+0000 7f0e9cc87700 1 -- 172.21.15.139:0/1893218121 wait complete. 2021-11-20T06:03:10.806 INFO:teuthology.orchestra.run.smithi139.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-11-20T06:03:10.808 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2021-11-20T06:03:10.808 INFO:tasks.cephadm:Setup complete, yielding 2021-11-20T06:03:10.809 INFO:teuthology.run_tasks:Running task print... 2021-11-20T06:03:10.828 INFO:teuthology.task.print:**** done starting v16.2.4 2021-11-20T06:03:10.829 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T06:03:11.006 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi139.front.sepia.ceph.com 2021-11-20T06:03:11.007 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch status' 2021-11-20T06:03:11.442 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:11 smithi139 conmon[33501]: cluster 2021-11-20T06:03:10.205839+0000 mgr.smithi139.dplyxn (mgr.14162) 170 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:11.442 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:11 smithi139 conmon[33501]: audit 2021-11-20T06: 2021-11-20T06:03:11.442 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:11 smithi139 conmon[33501]: 03:10.462422+0000 mon.smithi139 (mon.0) 443 : audit [DBG] from='client.? 172.21.15.139:0/1893218121' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-11-20T06:03:11.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:11 smithi178 conmon[36600]: cluster 2021-11-20T06:03:10.205839+0000 mgr.smithi139.dplyxn (mgr.14162) 170 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:11.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:11 smithi178 conmon[36600]: audit 2021-11-20T06:03:10.462422+0000 mon.smithi139 (mon.0) 443 : audit [DBG] from='client.? 172.21.15.139:0/1893218121' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-11-20T06:03:12.780 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.791+0000 7f4c79214700 1 -- 172.21.15.139:0/3941676824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c740f6860 msgr2=0x7f4c740f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:12.780 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.791+0000 7f4c79214700 1 --2- 172.21.15.139:0/3941676824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c740f6860 0x7f4c740f6cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f4c5c004660 tx=0x7f4c5c009b30).stop 2021-11-20T06:03:12.780 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.791+0000 7f4c79214700 1 -- 172.21.15.139:0/3941676824 shutdown_connections 2021-11-20T06:03:12.781 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.791+0000 7f4c79214700 1 --2- 172.21.15.139:0/3941676824 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c740f6860 0x7f4c740f6cc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:12.781 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.791+0000 7f4c79214700 1 --2- 172.21.15.139:0/3941676824 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4c740f55c0 0x7f4c740f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:12.781 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.791+0000 7f4c79214700 1 -- 172.21.15.139:0/3941676824 >> 172.21.15.139:0/3941676824 conn(0x7f4c740f0d40 msgr2=0x7f4c740f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:12.781 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.791+0000 7f4c79214700 1 -- 172.21.15.139:0/3941676824 shutdown_connections 2021-11-20T06:03:12.782 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.791+0000 7f4c79214700 1 -- 172.21.15.139:0/3941676824 wait complete. 2021-11-20T06:03:12.782 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.791+0000 7f4c79214700 1 Processor -- start 2021-11-20T06:03:12.782 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.791+0000 7f4c79214700 1 -- start start 2021-11-20T06:03:12.782 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c79214700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4c740f55c0 0x7f4c74104d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:12.783 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c79214700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c740f6860 0x7f4c74105200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:12.783 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c79214700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c74105800 con 0x7f4c740f6860 2021-11-20T06:03:12.783 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c79214700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c74105940 con 0x7f4c740f55c0 2021-11-20T06:03:12.783 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c7259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c740f6860 0x7f4c74105200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:12.783 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c7259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c740f6860 0x7f4c74105200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57322/0 (socket says 172.21.15.139:57322) 2021-11-20T06:03:12.784 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c7259c700 1 -- 172.21.15.139:0/3360130076 learned_addr learned my addr 172.21.15.139:0/3360130076 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:12.784 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c72d9d700 1 --2- 172.21.15.139:0/3360130076 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4c740f55c0 0x7f4c74104d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:12.785 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c7259c700 1 -- 172.21.15.139:0/3360130076 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4c740f55c0 msgr2=0x7f4c74104d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:12.785 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c7259c700 1 --2- 172.21.15.139:0/3360130076 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4c740f55c0 0x7f4c74104d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:12.785 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c7259c700 1 -- 172.21.15.139:0/3360130076 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c005040 con 0x7f4c740f6860 2021-11-20T06:03:12.785 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c7259c700 1 --2- 172.21.15.139:0/3360130076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c740f6860 0x7f4c74105200 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f4c5c009830 tx=0x7f4c5c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:12.786 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c6b7fe700 1 -- 172.21.15.139:0/3360130076 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c5c018070 con 0x7f4c740f6860 2021-11-20T06:03:12.786 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c79214700 1 -- 172.21.15.139:0/3360130076 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c74106250 con 0x7f4c740f6860 2021-11-20T06:03:12.786 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c6b7fe700 1 -- 172.21.15.139:0/3360130076 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c5c013410 con 0x7f4c740f6860 2021-11-20T06:03:12.786 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c6b7fe700 1 -- 172.21.15.139:0/3360130076 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c5c01c400 con 0x7f4c740f6860 2021-11-20T06:03:12.786 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.792+0000 7f4c79214700 1 -- 172.21.15.139:0/3360130076 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c741067b0 con 0x7f4c740f6860 2021-11-20T06:03:12.787 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.793+0000 7f4c79214700 1 -- 172.21.15.139:0/3360130076 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c74045ad0 con 0x7f4c740f6860 2021-11-20T06:03:12.788 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.794+0000 7f4c6b7fe700 1 -- 172.21.15.139:0/3360130076 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4c5c01d420 con 0x7f4c740f6860 2021-11-20T06:03:12.788 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.795+0000 7f4c6b7fe700 1 --2- 172.21.15.139:0/3360130076 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4c6005c2b0 0x7f4c6005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:12.788 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.795+0000 7f4c6b7fe700 1 -- 172.21.15.139:0/3360130076 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4c5c077e20 con 0x7f4c740f6860 2021-11-20T06:03:12.789 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.795+0000 7f4c72d9d700 1 --2- 172.21.15.139:0/3360130076 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4c6005c2b0 0x7f4c6005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:12.789 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.795+0000 7f4c72d9d700 1 --2- 172.21.15.139:0/3360130076 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4c6005c2b0 0x7f4c6005e770 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f4c74104aa0 tx=0x7f4c64009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:12.789 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.798+0000 7f4c6b7fe700 1 -- 172.21.15.139:0/3360130076 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c5c04aeb0 con 0x7f4c740f6860 2021-11-20T06:03:12.967 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.980+0000 7f4c79214700 1 -- 172.21.15.139:0/3360130076 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c740faaa0 con 0x7f4c6005c2b0 2021-11-20T06:03:12.968 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.981+0000 7f4c6b7fe700 1 -- 172.21.15.139:0/3360130076 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f4c740faaa0 con 0x7f4c6005c2b0 2021-11-20T06:03:12.969 INFO:teuthology.orchestra.run.smithi139.stdout:Backend: cephadm 2021-11-20T06:03:12.969 INFO:teuthology.orchestra.run.smithi139.stdout:Available: Yes 2021-11-20T06:03:12.970 INFO:teuthology.orchestra.run.smithi139.stdout:Paused: No 2021-11-20T06:03:12.972 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.983+0000 7f4c79214700 1 -- 172.21.15.139:0/3360130076 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4c6005c2b0 msgr2=0x7f4c6005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:12.973 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.983+0000 7f4c79214700 1 --2- 172.21.15.139:0/3360130076 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4c6005c2b0 0x7f4c6005e770 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f4c74104aa0 tx=0x7f4c64009040).stop 2021-11-20T06:03:12.974 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.983+0000 7f4c79214700 1 -- 172.21.15.139:0/3360130076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c740f6860 msgr2=0x7f4c74105200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:12.974 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.983+0000 7f4c79214700 1 --2- 172.21.15.139:0/3360130076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c740f6860 0x7f4c74105200 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f4c5c009830 tx=0x7f4c5c009b30).stop 2021-11-20T06:03:12.975 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.983+0000 7f4c79214700 1 -- 172.21.15.139:0/3360130076 shutdown_connections 2021-11-20T06:03:12.975 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.983+0000 7f4c79214700 1 --2- 172.21.15.139:0/3360130076 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f4c6005c2b0 0x7f4c6005e770 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:12.975 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.983+0000 7f4c79214700 1 --2- 172.21.15.139:0/3360130076 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4c740f55c0 0x7f4c74104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:12.976 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.983+0000 7f4c79214700 1 --2- 172.21.15.139:0/3360130076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c740f6860 0x7f4c74105200 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:12.976 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.983+0000 7f4c79214700 1 -- 172.21.15.139:0/3360130076 >> 172.21.15.139:0/3360130076 conn(0x7f4c740f0d40 msgr2=0x7f4c7405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:12.976 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.983+0000 7f4c79214700 1 -- 172.21.15.139:0/3360130076 shutdown_connections 2021-11-20T06:03:12.977 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:12.983+0000 7f4c79214700 1 -- 172.21.15.139:0/3360130076 wait complete. 2021-11-20T06:03:13.424 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch ps' 2021-11-20T06:03:14.420 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:14 smithi139 conmon[33501]: cluster 2021-11-20T06:03:12.206293+0000 mgr.smithi139.dplyxn (mgr.14162) 171 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:14.420 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:14 smithi139 conmon[33501]: audit 2021-11-20T06:03:12.983252+0000 mgr.smithi139.dplyxn (mgr.14162) 172 : audit [DBG] from='client.14390 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:03:14.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:14 smithi178 conmon[36600]: cluster 2021-11-20T06:03:12.206293+0000 mgr.smithi139.dplyxn (mgr.14162) 171 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:14.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:14 smithi178 conmon[36600]: audit 2021-11-20T06:03:12.983252+0000 mgr.smithi139.dplyxn (mgr.14162) 172 : audit [DBG] from='client.14390 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:03:14.751 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.762+0000 7f0d5b898700 1 -- 172.21.15.139:0/3114451040 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d540f6680 msgr2=0x7f0d540f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:14.751 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.762+0000 7f0d5b898700 1 --2- 172.21.15.139:0/3114451040 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d540f6680 0x7f0d540f6aa0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f0d50004660 tx=0x7f0d50009b30).stop 2021-11-20T06:03:14.752 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.762+0000 7f0d5b898700 1 -- 172.21.15.139:0/3114451040 shutdown_connections 2021-11-20T06:03:14.752 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.762+0000 7f0d5b898700 1 --2- 172.21.15.139:0/3114451040 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d540f7920 0x7f0d540f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:14.752 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.762+0000 7f0d5b898700 1 --2- 172.21.15.139:0/3114451040 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d540f6680 0x7f0d540f6aa0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:14.752 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.763+0000 7f0d5b898700 1 -- 172.21.15.139:0/3114451040 >> 172.21.15.139:0/3114451040 conn(0x7f0d540f1e00 msgr2=0x7f0d540f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:14.753 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.763+0000 7f0d5b898700 1 -- 172.21.15.139:0/3114451040 shutdown_connections 2021-11-20T06:03:14.753 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.763+0000 7f0d5b898700 1 -- 172.21.15.139:0/3114451040 wait complete. 2021-11-20T06:03:14.753 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.763+0000 7f0d5b898700 1 Processor -- start 2021-11-20T06:03:14.753 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.763+0000 7f0d5b898700 1 -- start start 2021-11-20T06:03:14.754 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.763+0000 7f0d5b898700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d540f6680 0x7f0d5406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:14.754 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.763+0000 7f0d5b898700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d540f7920 0x7f0d54069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:14.754 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.763+0000 7f0d5b898700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5406a380 con 0x7f0d540f6680 2021-11-20T06:03:14.754 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.763+0000 7f0d5b898700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5406a4c0 con 0x7f0d540f7920 2021-11-20T06:03:14.755 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.763+0000 7f0d59634700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d540f6680 0x7f0d5406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:14.755 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.763+0000 7f0d59634700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d540f6680 0x7f0d5406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57330/0 (socket says 172.21.15.139:57330) 2021-11-20T06:03:14.755 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.763+0000 7f0d59634700 1 -- 172.21.15.139:0/538200338 learned_addr learned my addr 172.21.15.139:0/538200338 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:14.755 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.764+0000 7f0d58e33700 1 --2- 172.21.15.139:0/538200338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d540f7920 0x7f0d54069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:14.755 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.764+0000 7f0d59634700 1 -- 172.21.15.139:0/538200338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d540f7920 msgr2=0x7f0d54069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:14.756 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.764+0000 7f0d59634700 1 --2- 172.21.15.139:0/538200338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d540f7920 0x7f0d54069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:14.756 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.764+0000 7f0d59634700 1 -- 172.21.15.139:0/538200338 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d50005040 con 0x7f0d540f6680 2021-11-20T06:03:14.756 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.764+0000 7f0d58e33700 1 --2- 172.21.15.139:0/538200338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d540f7920 0x7f0d54069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:03:14.756 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.764+0000 7f0d59634700 1 --2- 172.21.15.139:0/538200338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d540f6680 0x7f0d5406b9a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f0d50009f50 tx=0x7f0d50009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:14.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.764+0000 7f0d49ffb700 1 -- 172.21.15.139:0/538200338 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d50018070 con 0x7f0d540f6680 2021-11-20T06:03:14.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.764+0000 7f0d5b898700 1 -- 172.21.15.139:0/538200338 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5406a740 con 0x7f0d540f6680 2021-11-20T06:03:14.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.764+0000 7f0d49ffb700 1 -- 172.21.15.139:0/538200338 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d50013410 con 0x7f0d540f6680 2021-11-20T06:03:14.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.764+0000 7f0d49ffb700 1 -- 172.21.15.139:0/538200338 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d5001c620 con 0x7f0d540f6680 2021-11-20T06:03:14.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.764+0000 7f0d5b898700 1 -- 172.21.15.139:0/538200338 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5406aca0 con 0x7f0d540f6680 2021-11-20T06:03:14.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.765+0000 7f0d5b898700 1 -- 172.21.15.139:0/538200338 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d54045ad0 con 0x7f0d540f6680 2021-11-20T06:03:14.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.765+0000 7f0d49ffb700 1 -- 172.21.15.139:0/538200338 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0d5001d420 con 0x7f0d540f6680 2021-11-20T06:03:14.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.766+0000 7f0d49ffb700 1 --2- 172.21.15.139:0/538200338 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0d4005c300 0x7f0d4005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:14.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.766+0000 7f0d49ffb700 1 -- 172.21.15.139:0/538200338 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f0d50077970 con 0x7f0d540f6680 2021-11-20T06:03:14.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.766+0000 7f0d58e33700 1 --2- 172.21.15.139:0/538200338 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0d4005c300 0x7f0d4005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:14.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.766+0000 7f0d58e33700 1 --2- 172.21.15.139:0/538200338 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0d4005c300 0x7f0d4005e7c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f0d440041d0 tx=0x7f0d44009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:14.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.769+0000 7f0d49ffb700 1 -- 172.21.15.139:0/538200338 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d5004a9f0 con 0x7f0d540f6680 2021-11-20T06:03:14.933 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.947+0000 7f0d5b898700 1 -- 172.21.15.139:0/538200338 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d540fef50 con 0x7f0d4005c300 2021-11-20T06:03:14.940 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.954+0000 7f0d49ffb700 1 -- 172.21.15.139:0/538200338 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2160 (secure 0 0 0) 0x7f0d540fef50 con 0x7f0d4005c300 2021-11-20T06:03:14.941 INFO:teuthology.orchestra.run.smithi139.stdout:NAME HOST PORTS STATUS REFRESHED AGE VERSION IMAGE ID CONTAINER ID 2021-11-20T06:03:14.941 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager.smithi139 smithi139 *:9093,9094 running (2m) 41s ago 3m 0.20.0 0881eb8f169f 41cbc6fa19a3 2021-11-20T06:03:14.941 INFO:teuthology.orchestra.run.smithi139.stdout:crash.smithi139 smithi139 running (3m) 41s ago 3m 16.2.4 8d91d370c2b8 49a2ea91b9e1 2021-11-20T06:03:14.941 INFO:teuthology.orchestra.run.smithi139.stdout:crash.smithi178 smithi178 running (2m) 51s ago 2m 16.2.4 8d91d370c2b8 fdc8fe9f2cb7 2021-11-20T06:03:14.942 INFO:teuthology.orchestra.run.smithi139.stdout:grafana.smithi139 smithi139 *:3000 running (2m) 41s ago 3m 6.7.4 557c83e11646 e8c11a962df2 2021-11-20T06:03:14.942 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.smithi139.dplyxn smithi139 *:9283 running (4m) 41s ago 4m 16.2.4 8d91d370c2b8 50ebe8a684e5 2021-11-20T06:03:14.942 INFO:teuthology.orchestra.run.smithi139.stdout:mgr.smithi178.zxxdkz smithi178 *:8443,9283 running (2m) 51s ago 2m 16.2.4 8d91d370c2b8 1389d53683b3 2021-11-20T06:03:14.942 INFO:teuthology.orchestra.run.smithi139.stdout:mon.smithi139 smithi139 running (4m) 41s ago 4m 16.2.4 8d91d370c2b8 e8a76ef704b8 2021-11-20T06:03:14.942 INFO:teuthology.orchestra.run.smithi139.stdout:mon.smithi178 smithi178 running (2m) 51s ago 2m 16.2.4 8d91d370c2b8 3dba80a538b5 2021-11-20T06:03:14.943 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.smithi139 smithi139 *:9100 running (3m) 41s ago 3m 0.18.1 e5a616e4b9cf ecc922979a5e 2021-11-20T06:03:14.943 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter.smithi178 smithi178 *:9100 running (2m) 51s ago 2m 0.18.1 e5a616e4b9cf 2b8fd7c9dabf 2021-11-20T06:03:14.943 INFO:teuthology.orchestra.run.smithi139.stdout:osd.0 smithi139 running (2m) 41s ago 2m 16.2.4 8d91d370c2b8 657ff498d6c9 2021-11-20T06:03:14.943 INFO:teuthology.orchestra.run.smithi139.stdout:osd.1 smithi139 running (2m) 41s ago 2m 16.2.4 8d91d370c2b8 8d2e4d532c5e 2021-11-20T06:03:14.944 INFO:teuthology.orchestra.run.smithi139.stdout:osd.2 smithi139 running (103s) 41s ago 102s 16.2.4 8d91d370c2b8 fea61fa943fc 2021-11-20T06:03:14.944 INFO:teuthology.orchestra.run.smithi139.stdout:osd.3 smithi178 running (87s) 51s ago 86s 16.2.4 8d91d370c2b8 9a1ee4366489 2021-11-20T06:03:14.944 INFO:teuthology.orchestra.run.smithi139.stdout:osd.4 smithi178 running (71s) 51s ago 70s 16.2.4 8d91d370c2b8 7e7b18739aab 2021-11-20T06:03:14.944 INFO:teuthology.orchestra.run.smithi139.stdout:osd.5 smithi178 running (54s) 51s ago 54s 16.2.4 8d91d370c2b8 5d31fefb2ae6 2021-11-20T06:03:14.944 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus.smithi139 smithi139 *:9095 running (2m) 41s ago 3m 2.18.1 de242295e225 0239ac2001c2 2021-11-20T06:03:14.945 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.956+0000 7f0d5b898700 1 -- 172.21.15.139:0/538200338 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0d4005c300 msgr2=0x7f0d4005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:14.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.956+0000 7f0d5b898700 1 --2- 172.21.15.139:0/538200338 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0d4005c300 0x7f0d4005e7c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f0d440041d0 tx=0x7f0d44009040).stop 2021-11-20T06:03:14.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.956+0000 7f0d5b898700 1 -- 172.21.15.139:0/538200338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d540f6680 msgr2=0x7f0d5406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:14.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.956+0000 7f0d5b898700 1 --2- 172.21.15.139:0/538200338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d540f6680 0x7f0d5406b9a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f0d50009f50 tx=0x7f0d50009b30).stop 2021-11-20T06:03:14.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.956+0000 7f0d5b898700 1 -- 172.21.15.139:0/538200338 shutdown_connections 2021-11-20T06:03:14.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.956+0000 7f0d5b898700 1 --2- 172.21.15.139:0/538200338 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0d4005c300 0x7f0d4005e7c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:14.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.956+0000 7f0d5b898700 1 --2- 172.21.15.139:0/538200338 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d540f6680 0x7f0d5406b9a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:14.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.956+0000 7f0d5b898700 1 --2- 172.21.15.139:0/538200338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d540f7920 0x7f0d54069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:14.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.956+0000 7f0d5b898700 1 -- 172.21.15.139:0/538200338 >> 172.21.15.139:0/538200338 conn(0x7f0d540f1e00 msgr2=0x7f0d540fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:14.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.956+0000 7f0d5b898700 1 -- 172.21.15.139:0/538200338 shutdown_connections 2021-11-20T06:03:14.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:14.956+0000 7f0d5b898700 1 -- 172.21.15.139:0/538200338 wait complete. 2021-11-20T06:03:15.327 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch ls' 2021-11-20T06:03:15.352 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:15 smithi139 conmon[33501]: cluster 2021-11-20T06:03:14.206590+0000 mgr.smithi139.dplyxn (mgr.14162) 173 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:15.353 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:15 smithi139 conmon[33501]: audit 2021-11-20T06:03:14.949819+0000 mgr.smithi139.dplyxn (mgr.14162) 174 : audit [DBG] from='client.14394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:03:15.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:15 smithi178 conmon[36600]: cluster 2021-11-20T06:03:14.206590+0000 mgr.smithi139.dplyxn (mgr.14162) 173 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:15.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:15 smithi178 conmon[36600]: audit 2021-11-20T06:03:14.949819+0000 mgr.smithi139.dplyxn (mgr.14162) 174 : audit [DBG] from='client.14394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:03:16.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.887+0000 7f721186c700 1 -- 172.21.15.139:0/2473234776 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f720c0f7980 msgr2=0x7f720c0f7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:16.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.887+0000 7f721186c700 1 --2- 172.21.15.139:0/2473234776 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f720c0f7980 0x7f720c0f7de0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f71f4004660 tx=0x7f71f4009b30).stop 2021-11-20T06:03:16.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.887+0000 7f721186c700 1 -- 172.21.15.139:0/2473234776 shutdown_connections 2021-11-20T06:03:16.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.887+0000 7f721186c700 1 --2- 172.21.15.139:0/2473234776 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f720c0f7980 0x7f720c0f7de0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:16.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.887+0000 7f721186c700 1 --2- 172.21.15.139:0/2473234776 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f720c0f66e0 0x7f720c0f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:03:16.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.887+0000 7f721186c700 1 -- 172.21.15.139:0/2473234776 >> 172.21.15.139:0/2473234776 conn(0x7f720c0f1e40 msgr2=0x7f720c0f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:16.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.887+0000 7f721186c700 1 -- 172.21.15.139:0/2473234776 shutdown_connections 2021-11-20T06:03:16.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.887+0000 7f721186c700 1 -- 172.21.15.139:0/2473234776 wait complete. 2021-11-20T06:03:16.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.887+0000 7f721186c700 1 Processor -- start 2021-11-20T06:03:16.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.887+0000 7f721186c700 1 -- start start 2021-11-20T06:03:16.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.887+0000 7f721186c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f720c0f66e0 0x7f720c106fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:16.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.887+0000 7f721186c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f720c0f7980 0x7f720c1074c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:16.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.887+0000 7f721186c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720c107b00 con 0x7f720c0f7980 2021-11-20T06:03:16.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.888+0000 7f721186c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720c1083b0 con 0x7f720c0f66e0 2021-11-20T06:03:16.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.888+0000 7f720a7fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f720c0f7980 0x7f720c1074c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:16.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.888+0000 7f720a7fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f720c0f7980 0x7f720c1074c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57342/0 (socket says 172.21.15.139:57342) 2021-11-20T06:03:16.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.888+0000 7f720a7fc700 1 -- 172.21.15.139:0/2447210781 learned_addr learned my addr 172.21.15.139:0/2447210781 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:16.880 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.888+0000 7f720affd700 1 --2- 172.21.15.139:0/2447210781 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f720c0f66e0 0x7f720c106fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:16.880 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.888+0000 7f720a7fc700 1 -- 172.21.15.139:0/2447210781 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f720c0f66e0 msgr2=0x7f720c106fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:16.880 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.888+0000 7f720a7fc700 1 --2- 172.21.15.139:0/2447210781 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f720c0f66e0 0x7f720c106fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:16.880 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.888+0000 7f720a7fc700 1 -- 172.21.15.139:0/2447210781 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f4005040 con 0x7f720c0f7980 2021-11-20T06:03:16.880 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.888+0000 7f720a7fc700 1 --2- 172.21.15.139:0/2447210781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f720c0f7980 0x7f720c1074c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f71f4009830 tx=0x7f71f4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:16.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.888+0000 7f72037fe700 1 -- 172.21.15.139:0/2447210781 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71f4018070 con 0x7f720c0f7980 2021-11-20T06:03:16.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.888+0000 7f72037fe700 1 -- 172.21.15.139:0/2447210781 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71f4013410 con 0x7f720c0f7980 2021-11-20T06:03:16.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.888+0000 7f72037fe700 1 -- 172.21.15.139:0/2447210781 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71f401c400 con 0x7f720c0f7980 2021-11-20T06:03:16.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.888+0000 7f721186c700 1 -- 172.21.15.139:0/2447210781 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f720c108630 con 0x7f720c0f7980 2021-11-20T06:03:16.882 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.888+0000 7f721186c700 1 -- 172.21.15.139:0/2447210781 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f720c108b40 con 0x7f720c0f7980 2021-11-20T06:03:16.882 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.889+0000 7f721186c700 1 -- 172.21.15.139:0/2447210781 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f720c0f5ad0 con 0x7f720c0f7980 2021-11-20T06:03:16.883 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.890+0000 7f72037fe700 1 -- 172.21.15.139:0/2447210781 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f71f401d420 con 0x7f720c0f7980 2021-11-20T06:03:16.884 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.890+0000 7f72037fe700 1 --2- 172.21.15.139:0/2447210781 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f71f805c2b0 0x7f71f805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:16.884 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.890+0000 7f72037fe700 1 -- 172.21.15.139:0/2447210781 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f71f4077a50 con 0x7f720c0f7980 2021-11-20T06:03:16.884 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.890+0000 7f720affd700 1 --2- 172.21.15.139:0/2447210781 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f71f805c2b0 0x7f71f805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:16.884 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.890+0000 7f720affd700 1 --2- 172.21.15.139:0/2447210781 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f71f805c2b0 0x7f71f805e770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f720c0f77e0 tx=0x7f71fc009220).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:16.885 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:16.894+0000 7f72037fe700 1 -- 172.21.15.139:0/2447210781 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71f404a9f0 con 0x7f720c0f7980 2021-11-20T06:03:17.053 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:17.066+0000 7f721186c700 1 -- 172.21.15.139:0/2447210781 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f720c0fef80 con 0x7f71f805c2b0 2021-11-20T06:03:17.057 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:17.070+0000 7f72037fe700 1 -- 172.21.15.139:0/2447210781 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1125 (secure 0 0 0) 0x7f720c0fef80 con 0x7f71f805c2b0 2021-11-20T06:03:17.057 INFO:teuthology.orchestra.run.smithi139.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2021-11-20T06:03:17.057 INFO:teuthology.orchestra.run.smithi139.stdout:alertmanager 1/1 43s ago 3m count:1 2021-11-20T06:03:17.057 INFO:teuthology.orchestra.run.smithi139.stdout:crash 2/2 53s ago 4m * 2021-11-20T06:03:17.058 INFO:teuthology.orchestra.run.smithi139.stdout:grafana 1/1 43s ago 4m count:1 2021-11-20T06:03:17.058 INFO:teuthology.orchestra.run.smithi139.stdout:mgr 2/2 53s ago 4m count:2 2021-11-20T06:03:17.058 INFO:teuthology.orchestra.run.smithi139.stdout:mon 2/2 53s ago 3m smithi139:172.21.15.139=smithi139;smithi178:172.21.15.178=smithi178;count:2 2021-11-20T06:03:17.058 INFO:teuthology.orchestra.run.smithi139.stdout:node-exporter 2/2 53s ago 4m * 2021-11-20T06:03:17.059 INFO:teuthology.orchestra.run.smithi139.stdout:osd.unmanaged 6/6 53s ago - 2021-11-20T06:03:17.059 INFO:teuthology.orchestra.run.smithi139.stdout:prometheus 1/1 43s ago 4m count:1 2021-11-20T06:03:17.060 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:17.072+0000 7f721186c700 1 -- 172.21.15.139:0/2447210781 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f71f805c2b0 msgr2=0x7f71f805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:17.060 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:17.072+0000 7f721186c700 1 --2- 172.21.15.139:0/2447210781 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f71f805c2b0 0x7f71f805e770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f720c0f77e0 tx=0x7f71fc009220).stop 2021-11-20T06:03:17.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:17.072+0000 7f721186c700 1 -- 172.21.15.139:0/2447210781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f720c0f7980 msgr2=0x7f720c1074c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:17.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:17.072+0000 7f721186c700 1 --2- 172.21.15.139:0/2447210781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f720c0f7980 0x7f720c1074c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f71f4009830 tx=0x7f71f4009b30).stop 2021-11-20T06:03:17.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:17.072+0000 7f721186c700 1 -- 172.21.15.139:0/2447210781 shutdown_connections 2021-11-20T06:03:17.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:17.072+0000 7f721186c700 1 --2- 172.21.15.139:0/2447210781 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f71f805c2b0 0x7f71f805e770 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:17.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:17.072+0000 7f721186c700 1 --2- 172.21.15.139:0/2447210781 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f720c0f66e0 0x7f720c106fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:17.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:17.072+0000 7f721186c700 1 --2- 172.21.15.139:0/2447210781 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f720c0f7980 0x7f720c1074c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:17.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:17.072+0000 7f721186c700 1 -- 172.21.15.139:0/2447210781 >> 172.21.15.139:0/2447210781 conn(0x7f720c0f1e40 msgr2=0x7f720c0fb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:17.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:17.072+0000 7f721186c700 1 -- 172.21.15.139:0/2447210781 shutdown_connections 2021-11-20T06:03:17.063 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:17.072+0000 7f721186c700 1 -- 172.21.15.139:0/2447210781 wait complete. 2021-11-20T06:03:17.441 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch host ls' 2021-11-20T06:03:18.277 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:18 smithi139 conmon[33501]: cluster 2021-11-20T06:03:16.206892+0000 mgr.smithi139.dplyxn (mgr.14162) 175 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:18.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:18 smithi139 conmon[33501]: audit 2021-11-20T06:03:17.069007+0000 mgr.smithi139.dplyxn (mgr.14162) 176 : audit [DBG] from='client.14398 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:03:18.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:18 smithi178 conmon[36600]: cluster 2021-11-20T06:03:16.206892+0000 mgr.smithi139.dplyxn (mgr.14162) 175 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:18.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:18 smithi178 conmon[36600]: audit 2021-11-20T06:03:17.069007+0000 mgr.smithi139.dplyxn (mgr.14162) 176 : audit [DBG] from='client.14398 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:03:19.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.148+0000 7f15cf249700 1 -- 172.21.15.139:0/3913133428 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15c80f5ed0 msgr2=0x7f15c80f6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:19.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.148+0000 7f15cf249700 1 --2- 172.21.15.139:0/3913133428 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15c80f5ed0 0x7f15c80f6900 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f15b4004660 tx=0x7f15b4009b30).stop 2021-11-20T06:03:19.139 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.148+0000 7f15cf249700 1 -- 172.21.15.139:0/3913133428 shutdown_connections 2021-11-20T06:03:19.139 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.148+0000 7f15cf249700 1 --2- 172.21.15.139:0/3913133428 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15c80f5ed0 0x7f15c80f6900 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:19.139 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.148+0000 7f15cf249700 1 --2- 172.21.15.139:0/3913133428 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15c80f55c0 0x7f15c80f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:03:19.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.148+0000 7f15cf249700 1 -- 172.21.15.139:0/3913133428 >> 172.21.15.139:0/3913133428 conn(0x7f15c80f0de0 msgr2=0x7f15c80f3200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:19.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.149+0000 7f15cf249700 1 -- 172.21.15.139:0/3913133428 shutdown_connections 2021-11-20T06:03:19.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.149+0000 7f15cf249700 1 -- 172.21.15.139:0/3913133428 wait complete. 2021-11-20T06:03:19.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.149+0000 7f15cf249700 1 Processor -- start 2021-11-20T06:03:19.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.149+0000 7f15cf249700 1 -- start start 2021-11-20T06:03:19.141 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.150+0000 7f15cf249700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15c80f55c0 0x7f15c8105e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:19.142 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.150+0000 7f15cf249700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15c80f5ed0 0x7f15c8106380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:19.143 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.150+0000 7f15cf249700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c81069c0 con 0x7f15c80f55c0 2021-11-20T06:03:19.143 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.150+0000 7f15cf249700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c8107270 con 0x7f15c80f5ed0 2021-11-20T06:03:19.143 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.150+0000 7f15ccfe5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15c80f55c0 0x7f15c8105e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:19.143 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.150+0000 7f15ccfe5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15c80f55c0 0x7f15c8105e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57350/0 (socket says 172.21.15.139:57350) 2021-11-20T06:03:19.144 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.150+0000 7f15ccfe5700 1 -- 172.21.15.139:0/1210370361 learned_addr learned my addr 172.21.15.139:0/1210370361 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:19.144 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.150+0000 7f15c7fff700 1 --2- 172.21.15.139:0/1210370361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15c80f5ed0 0x7f15c8106380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:19.144 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.150+0000 7f15ccfe5700 1 -- 172.21.15.139:0/1210370361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15c80f5ed0 msgr2=0x7f15c8106380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:19.144 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.150+0000 7f15ccfe5700 1 --2- 172.21.15.139:0/1210370361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15c80f5ed0 0x7f15c8106380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:19.145 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.150+0000 7f15ccfe5700 1 -- 172.21.15.139:0/1210370361 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b4005040 con 0x7f15c80f55c0 2021-11-20T06:03:19.145 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.150+0000 7f15ccfe5700 1 --2- 172.21.15.139:0/1210370361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15c80f55c0 0x7f15c8105e90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f15bc00b410 tx=0x7f15bc006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:19.145 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.151+0000 7f15c57fa700 1 -- 172.21.15.139:0/1210370361 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15bc0125f0 con 0x7f15c80f55c0 2021-11-20T06:03:19.145 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.151+0000 7f15cf249700 1 -- 172.21.15.139:0/1210370361 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c8107550 con 0x7f15c80f55c0 2021-11-20T06:03:19.145 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.151+0000 7f15c57fa700 1 -- 172.21.15.139:0/1210370361 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15bc00be30 con 0x7f15c80f55c0 2021-11-20T06:03:19.146 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.151+0000 7f15c57fa700 1 -- 172.21.15.139:0/1210370361 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15bc002a20 con 0x7f15c80f55c0 2021-11-20T06:03:19.146 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.151+0000 7f15cf249700 1 -- 172.21.15.139:0/1210370361 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c8107b10 con 0x7f15c80f55c0 2021-11-20T06:03:19.146 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.151+0000 7f15cf249700 1 -- 172.21.15.139:0/1210370361 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c80f4a50 con 0x7f15c80f55c0 2021-11-20T06:03:19.147 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.152+0000 7f15c57fa700 1 -- 172.21.15.139:0/1210370361 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f15bc00f030 con 0x7f15c80f55c0 2021-11-20T06:03:19.148 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.152+0000 7f15c57fa700 1 --2- 172.21.15.139:0/1210370361 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f15b805c2b0 0x7f15b805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:19.148 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.153+0000 7f15c57fa700 1 -- 172.21.15.139:0/1210370361 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f15bc076330 con 0x7f15c80f55c0 2021-11-20T06:03:19.148 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.153+0000 7f15c7fff700 1 --2- 172.21.15.139:0/1210370361 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f15b805c2b0 0x7f15b805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:19.148 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.153+0000 7f15c7fff700 1 --2- 172.21.15.139:0/1210370361 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f15b805c2b0 0x7f15b805e770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f15b4004630 tx=0x7f15b4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:19.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.156+0000 7f15c57fa700 1 -- 172.21.15.139:0/1210370361 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15bc049420 con 0x7f15c80f55c0 2021-11-20T06:03:19.323 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.336+0000 7f15cf249700 1 -- 172.21.15.139:0/1210370361 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f15c80008d0 con 0x7f15b805c2b0 2021-11-20T06:03:19.323 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.337+0000 7f15c57fa700 1 -- 172.21.15.139:0/1210370361 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+129 (secure 0 0 0) 0x7f15c80008d0 con 0x7f15b805c2b0 2021-11-20T06:03:19.324 INFO:teuthology.orchestra.run.smithi139.stdout:HOST ADDR LABELS STATUS 2021-11-20T06:03:19.324 INFO:teuthology.orchestra.run.smithi139.stdout:smithi139 172.21.15.139 2021-11-20T06:03:19.324 INFO:teuthology.orchestra.run.smithi139.stdout:smithi178 smithi178 2021-11-20T06:03:19.326 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.339+0000 7f15cf249700 1 -- 172.21.15.139:0/1210370361 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f15b805c2b0 msgr2=0x7f15b805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:19.326 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.339+0000 7f15cf249700 1 --2- 172.21.15.139:0/1210370361 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f15b805c2b0 0x7f15b805e770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f15b4004630 tx=0x7f15b4009b30).stop 2021-11-20T06:03:19.326 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.339+0000 7f15cf249700 1 -- 172.21.15.139:0/1210370361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15c80f55c0 msgr2=0x7f15c8105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:19.326 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.339+0000 7f15cf249700 1 --2- 172.21.15.139:0/1210370361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15c80f55c0 0x7f15c8105e90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f15bc00b410 tx=0x7f15bc006ef0).stop 2021-11-20T06:03:19.327 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.339+0000 7f15cf249700 1 -- 172.21.15.139:0/1210370361 shutdown_connections 2021-11-20T06:03:19.327 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.339+0000 7f15cf249700 1 --2- 172.21.15.139:0/1210370361 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f15b805c2b0 0x7f15b805e770 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:19.327 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.339+0000 7f15cf249700 1 --2- 172.21.15.139:0/1210370361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15c80f55c0 0x7f15c8105e90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:19.327 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.339+0000 7f15cf249700 1 --2- 172.21.15.139:0/1210370361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15c80f5ed0 0x7f15c8106380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:19.328 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.339+0000 7f15cf249700 1 -- 172.21.15.139:0/1210370361 >> 172.21.15.139:0/1210370361 conn(0x7f15c80f0de0 msgr2=0x7f15c80fa350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:19.328 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.339+0000 7f15cf249700 1 -- 172.21.15.139:0/1210370361 shutdown_connections 2021-11-20T06:03:19.328 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:19.339+0000 7f15cf249700 1 -- 172.21.15.139:0/1210370361 wait complete. 2021-11-20T06:03:19.784 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch device ls' 2021-11-20T06:03:20.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:20 smithi178 conmon[36600]: cluster 2021-11-20T06:03:18.207170+0000 mgr.smithi139.dplyxn (mgr.14162) 177 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:20.470 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:20 smithi139 conmon[33501]: cluster 2021-11-20T06:03:18.207170+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:03:20.470 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:20 smithi139 conmon[33501]: 14162) 177 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:21.416 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.428+0000 7fca6e24f700 1 -- 172.21.15.139:0/802004572 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca680f49a0 msgr2=0x7fca680f4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:21.416 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.428+0000 7fca6e24f700 1 --2- 172.21.15.139:0/802004572 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca680f49a0 0x7fca680f4dc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fca58004660 tx=0x7fca58009b30).stop 2021-11-20T06:03:21.416 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.428+0000 7fca6e24f700 1 -- 172.21.15.139:0/802004572 shutdown_connections 2021-11-20T06:03:21.417 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.428+0000 7fca6e24f700 1 --2- 172.21.15.139:0/802004572 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca680f5340 0x7fca680f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:03:21.417 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.428+0000 7fca6e24f700 1 --2- 172.21.15.139:0/802004572 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca680f49a0 0x7fca680f4dc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:21.417 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.428+0000 7fca6e24f700 1 -- 172.21.15.139:0/802004572 >> 172.21.15.139:0/802004572 conn(0x7fca680f04c0 msgr2=0x7fca680f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:21.417 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.428+0000 7fca6e24f700 1 -- 172.21.15.139:0/802004572 shutdown_connections 2021-11-20T06:03:21.418 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.428+0000 7fca6e24f700 1 -- 172.21.15.139:0/802004572 wait complete. 2021-11-20T06:03:21.418 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.428+0000 7fca6e24f700 1 Processor -- start 2021-11-20T06:03:21.418 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca6e24f700 1 -- start start 2021-11-20T06:03:21.418 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca6e24f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca680f49a0 0x7fca68107080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:21.419 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca6e24f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca680f5340 0x7fca68107570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:21.419 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca6e24f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca68107bb0 con 0x7fca680f5340 2021-11-20T06:03:21.419 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca6e24f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca68108460 con 0x7fca680f49a0 2021-11-20T06:03:21.419 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca66ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca680f5340 0x7fca68107570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:21.419 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca66ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca680f5340 0x7fca68107570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57362/0 (socket says 172.21.15.139:57362) 2021-11-20T06:03:21.420 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca66ffd700 1 -- 172.21.15.139:0/3257172373 learned_addr learned my addr 172.21.15.139:0/3257172373 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:21.420 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca677fe700 1 --2- 172.21.15.139:0/3257172373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca680f49a0 0x7fca68107080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:21.420 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca66ffd700 1 -- 172.21.15.139:0/3257172373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca680f49a0 msgr2=0x7fca68107080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:21.420 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca66ffd700 1 --2- 172.21.15.139:0/3257172373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca680f49a0 0x7fca68107080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:21.421 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca66ffd700 1 -- 172.21.15.139:0/3257172373 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca58005040 con 0x7fca680f5340 2021-11-20T06:03:21.421 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca677fe700 1 --2- 172.21.15.139:0/3257172373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca680f49a0 0x7fca68107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:03:21.421 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:21 smithi139 conmon[33501]: audit 2021-11-20T06:03:19.338931+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:03:21.421 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:21 smithi139 conmon[33501]: ) 178 : audit [DBG] from='client.14402 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:03:21.422 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:21 smithi139 conmon[33501]: cluster 2021-11-20T06:03:20.207432+0000 mgr.smithi139.dplyxn (mgr.14162) 179 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:21.425 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca66ffd700 1 --2- 172.21.15.139:0/3257172373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca680f5340 0x7fca68107570 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fca5c0093f0 tx=0x7fca5c006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:21.425 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca4ffff700 1 -- 172.21.15.139:0/3257172373 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca5c009ea0 con 0x7fca680f5340 2021-11-20T06:03:21.425 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca6e24f700 1 -- 172.21.15.139:0/3257172373 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca68108740 con 0x7fca680f5340 2021-11-20T06:03:21.426 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca4ffff700 1 -- 172.21.15.139:0/3257172373 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca5c013af0 con 0x7fca680f5340 2021-11-20T06:03:21.426 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.429+0000 7fca4ffff700 1 -- 172.21.15.139:0/3257172373 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca5c00dad0 con 0x7fca680f5340 2021-11-20T06:03:21.426 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.430+0000 7fca6e24f700 1 -- 172.21.15.139:0/3257172373 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca68108c80 con 0x7fca680f5340 2021-11-20T06:03:21.426 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.431+0000 7fca4ffff700 1 -- 172.21.15.139:0/3257172373 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fca5c01a060 con 0x7fca680f5340 2021-11-20T06:03:21.426 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.431+0000 7fca6e24f700 1 -- 172.21.15.139:0/3257172373 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca68045ad0 con 0x7fca680f5340 2021-11-20T06:03:21.427 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.431+0000 7fca4ffff700 1 --2- 172.21.15.139:0/3257172373 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fca500606c0 0x7fca50062b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:21.427 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.431+0000 7fca4ffff700 1 -- 172.21.15.139:0/3257172373 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fca5c078220 con 0x7fca680f5340 2021-11-20T06:03:21.427 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.432+0000 7fca677fe700 1 --2- 172.21.15.139:0/3257172373 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fca500606c0 0x7fca50062b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:21.427 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.432+0000 7fca677fe700 1 --2- 172.21.15.139:0/3257172373 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fca500606c0 0x7fca50062b80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fca58004630 tx=0x7fca58009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:21.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.436+0000 7fca4ffff700 1 -- 172.21.15.139:0/3257172373 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca5c048190 con 0x7fca680f5340 2021-11-20T06:03:21.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:21 smithi178 conmon[36600]: audit 2021-11-20T06:03:19.338931+0000 mgr.smithi139.dplyxn (mgr.14162) 178 : audit [DBG] from='client.14402 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:03:21.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:21 smithi178 conmon[36600]: cluster 2021-11-20T06:03:20.207432+0000 mgr.smithi139.dplyxn (mgr.14162) 179 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:21.596 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.610+0000 7fca6e24f700 1 -- 172.21.15.139:0/3257172373 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fca680ff0e0 con 0x7fca500606c0 2021-11-20T06:03:21.597 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.611+0000 7fca4ffff700 1 -- 172.21.15.139:0/3257172373 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+279 (secure 0 0 0) 0x7fca58013860 con 0x7fca500606c0 2021-11-20T06:03:21.598 INFO:teuthology.orchestra.run.smithi139.stdout:Hostname Path Type Serial Size Health Ident Fault Available 2021-11-20T06:03:21.598 INFO:teuthology.orchestra.run.smithi139.stdout:smithi139 /dev/nvme0n1 ssd PHFT620500N6400BGN 400G Unknown N/A N/A No 2021-11-20T06:03:21.599 INFO:teuthology.orchestra.run.smithi139.stdout:smithi178 /dev/nvme0n1 ssd PHMB75130029480DGN 480G Unknown N/A N/A No 2021-11-20T06:03:21.600 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.613+0000 7fca6e24f700 1 -- 172.21.15.139:0/3257172373 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fca500606c0 msgr2=0x7fca50062b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:21.600 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.613+0000 7fca6e24f700 1 --2- 172.21.15.139:0/3257172373 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fca500606c0 0x7fca50062b80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fca58004630 tx=0x7fca58009b30).stop 2021-11-20T06:03:21.600 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.613+0000 7fca6e24f700 1 -- 172.21.15.139:0/3257172373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca680f5340 msgr2=0x7fca68107570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:21.600 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.613+0000 7fca6e24f700 1 --2- 172.21.15.139:0/3257172373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca680f5340 0x7fca68107570 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fca5c0093f0 tx=0x7fca5c006ec0).stop 2021-11-20T06:03:21.601 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.613+0000 7fca6e24f700 1 -- 172.21.15.139:0/3257172373 shutdown_connections 2021-11-20T06:03:21.601 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.613+0000 7fca6e24f700 1 --2- 172.21.15.139:0/3257172373 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fca500606c0 0x7fca50062b80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:21.601 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.613+0000 7fca6e24f700 1 --2- 172.21.15.139:0/3257172373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca680f49a0 0x7fca68107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:21.601 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.613+0000 7fca6e24f700 1 --2- 172.21.15.139:0/3257172373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fca680f5340 0x7fca68107570 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:21.602 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.613+0000 7fca6e24f700 1 -- 172.21.15.139:0/3257172373 >> 172.21.15.139:0/3257172373 conn(0x7fca680f04c0 msgr2=0x7fca680fb510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:21.602 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.613+0000 7fca6e24f700 1 -- 172.21.15.139:0/3257172373 shutdown_connections 2021-11-20T06:03:21.602 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:21.613+0000 7fca6e24f700 1 -- 172.21.15.139:0/3257172373 wait complete. 2021-11-20T06:03:21.986 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T06:03:21.997 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi139.front.sepia.ceph.com 2021-11-20T06:03:21.998 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs volume create cephfs --placement=4' 2021-11-20T06:03:23.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:23 smithi139 conmon[33501]: audit 2021-11-20T06:03:21.612468+0000 mgr.smithi139.dplyxn (mgr.14162) 180 : audit [DBG] from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:03:23.432 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.444+0000 7f30f4d4c700 1 -- 172.21.15.139:0/2379857902 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30f00f66e0 msgr2=0x7f30f00f6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:23.432 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.444+0000 7f30f4d4c700 1 --2- 172.21.15.139:0/2379857902 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30f00f66e0 0x7f30f00f6b40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f30e4004660 tx=0x7f30e4009b30).stop 2021-11-20T06:03:23.432 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.444+0000 7f30f4d4c700 1 -- 172.21.15.139:0/2379857902 shutdown_connections 2021-11-20T06:03:23.433 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.444+0000 7f30f4d4c700 1 --2- 172.21.15.139:0/2379857902 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30f00f66e0 0x7f30f00f6b40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:23.433 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.444+0000 7f30f4d4c700 1 --2- 172.21.15.139:0/2379857902 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30f00f5440 0x7f30f00f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:23.433 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.444+0000 7f30f4d4c700 1 -- 172.21.15.139:0/2379857902 >> 172.21.15.139:0/2379857902 conn(0x7f30f00f0ba0 msgr2=0x7f30f00f2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:23.433 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.444+0000 7f30f4d4c700 1 -- 172.21.15.139:0/2379857902 shutdown_connections 2021-11-20T06:03:23.434 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.444+0000 7f30f4d4c700 1 -- 172.21.15.139:0/2379857902 wait complete. 2021-11-20T06:03:23.434 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.445+0000 7f30f4d4c700 1 Processor -- start 2021-11-20T06:03:23.434 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.445+0000 7f30f4d4c700 1 -- start start 2021-11-20T06:03:23.434 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.445+0000 7f30f4d4c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30f00f5440 0x7f30f010beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:23.434 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.445+0000 7f30f4d4c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30f00f66e0 0x7f30f010e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:23.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.445+0000 7f30f4d4c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f00602d0 con 0x7f30f00f66e0 2021-11-20T06:03:23.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.445+0000 7f30f4d4c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f0060410 con 0x7f30f00f5440 2021-11-20T06:03:23.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.445+0000 7f30eeffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30f00f66e0 0x7f30f010e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:23.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.445+0000 7f30eeffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30f00f66e0 0x7f30f010e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57372/0 (socket says 172.21.15.139:57372) 2021-11-20T06:03:23.436 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.445+0000 7f30eeffd700 1 -- 172.21.15.139:0/2808683493 learned_addr learned my addr 172.21.15.139:0/2808683493 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:23.436 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.445+0000 7f30ef7fe700 1 --2- 172.21.15.139:0/2808683493 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30f00f5440 0x7f30f010beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:23.436 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.446+0000 7f30eeffd700 1 -- 172.21.15.139:0/2808683493 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30f00f5440 msgr2=0x7f30f010beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:23.436 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.446+0000 7f30eeffd700 1 --2- 172.21.15.139:0/2808683493 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30f00f5440 0x7f30f010beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:23.436 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.446+0000 7f30eeffd700 1 -- 172.21.15.139:0/2808683493 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e4005040 con 0x7f30f00f66e0 2021-11-20T06:03:23.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.446+0000 7f30eeffd700 1 --2- 172.21.15.139:0/2808683493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30f00f66e0 0x7f30f010e3b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f30e4009f50 tx=0x7f30e4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:23.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.446+0000 7f30d7fff700 1 -- 172.21.15.139:0/2808683493 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30e4018070 con 0x7f30f00f66e0 2021-11-20T06:03:23.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.446+0000 7f30f4d4c700 1 -- 172.21.15.139:0/2808683493 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f010e8a0 con 0x7f30f00f66e0 2021-11-20T06:03:23.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.446+0000 7f30d7fff700 1 -- 172.21.15.139:0/2808683493 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30e4013410 con 0x7f30f00f66e0 2021-11-20T06:03:23.438 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.446+0000 7f30d7fff700 1 -- 172.21.15.139:0/2808683493 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30e401c3c0 con 0x7f30f00f66e0 2021-11-20T06:03:23.438 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.446+0000 7f30f4d4c700 1 -- 172.21.15.139:0/2808683493 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f010ec70 con 0x7f30f00f66e0 2021-11-20T06:03:23.439 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.447+0000 7f30f4d4c700 1 -- 172.21.15.139:0/2808683493 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f0045ad0 con 0x7f30f00f66e0 2021-11-20T06:03:23.439 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.448+0000 7f30d7fff700 1 -- 172.21.15.139:0/2808683493 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f30e401d420 con 0x7f30f00f66e0 2021-11-20T06:03:23.440 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.448+0000 7f30d7fff700 1 --2- 172.21.15.139:0/2808683493 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f30d805c300 0x7f30d805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:23.440 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.448+0000 7f30d7fff700 1 -- 172.21.15.139:0/2808683493 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f30e4077ea0 con 0x7f30f00f66e0 2021-11-20T06:03:23.440 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.448+0000 7f30ef7fe700 1 --2- 172.21.15.139:0/2808683493 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f30d805c300 0x7f30d805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:23.441 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.449+0000 7f30ef7fe700 1 --2- 172.21.15.139:0/2808683493 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f30d805c300 0x7f30d805e7c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f30f0060060 tx=0x7f30e0009c00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:23.441 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.452+0000 7f30d7fff700 1 -- 172.21.15.139:0/2808683493 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30e404af90 con 0x7f30f00f66e0 2021-11-20T06:03:23.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:23 smithi178 conmon[36600]: audit 2021-11-20T06:03:21.612468+0000 mgr.smithi139.dplyxn (mgr.14162) 180 : audit [DBG] from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:03:23.610 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:23.625+0000 7f30f4d4c700 1 -- 172.21.15.139:0/2808683493 --> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f30f0057f10 con 0x7f30d805c300 2021-11-20T06:03:24.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:24 smithi178 conmon[36600]: cluster 2021-11-20T06:03:22.207716+0000 mgr.smithi139.dplyxn (mgr.14162) 181 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:24.518 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:24 smithi178 conmon[36600]: audit 2021-11-20T06:03:23.626600+0000 mgr.smithi139.dplyxn (mgr.14162) 182 : audit [DBG] from='client.14410 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:03:24.519 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:24 smithi178 conmon[36600]: audit 2021-11-20 2021-11-20T06:03:24.519 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:24 smithi178 conmon[36600]: T06:03:23.627092+0000 mon.smithi139 (mon.0) 444 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2021-11-20T06:03:24.519 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:24 smithi178 conmon[36600]: 2021-11-20T06:03:24.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:24 smithi139 conmon[33501]: cluster 2021-11-20T06:03:22.207716+0000 mgr.smithi139.dplyxn (mgr.14162) 181 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:24.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:24 smithi139 conmon[33501]: audit 2021-11-20T06:03:23.626600+0000 mgr.smithi139.dplyxn (mgr.14162) 182 : audit [DBG] from='client.14410 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-20T06:03:24.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:24 smithi139 conmon[33501]: audit 2021-11-20T06:03:23.627092+0000 mon.smithi139 (mon.0) 444 : audit 2021-11-20T06:03:24.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:24 smithi139 conmon[33501]: [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2021-11-20T06:03:25.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:25.223+0000 7f30d7fff700 1 -- 172.21.15.139:0/2808683493 <== mgr.14162 v2:172.21.15.139:6800/2363962993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f30f0057f10 con 0x7f30d805c300 2021-11-20T06:03:25.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:25.226+0000 7f30f4d4c700 1 -- 172.21.15.139:0/2808683493 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f30d805c300 msgr2=0x7f30d805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:25.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:25.226+0000 7f30f4d4c700 1 --2- 172.21.15.139:0/2808683493 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f30d805c300 0x7f30d805e7c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f30f0060060 tx=0x7f30e0009c00).stop 2021-11-20T06:03:25.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:25.226+0000 7f30f4d4c700 1 -- 172.21.15.139:0/2808683493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30f00f66e0 msgr2=0x7f30f010e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:25.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:25.226+0000 7f30f4d4c700 1 --2- 172.21.15.139:0/2808683493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30f00f66e0 0x7f30f010e3b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f30e4009f50 tx=0x7f30e4009b30).stop 2021-11-20T06:03:25.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:25.226+0000 7f30f4d4c700 1 -- 172.21.15.139:0/2808683493 shutdown_connections 2021-11-20T06:03:25.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:25.226+0000 7f30f4d4c700 1 --2- 172.21.15.139:0/2808683493 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f30d805c300 0x7f30d805e7c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:25.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:25.226+0000 7f30f4d4c700 1 --2- 172.21.15.139:0/2808683493 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30f00f5440 0x7f30f010beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:25.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:25.226+0000 7f30f4d4c700 1 --2- 172.21.15.139:0/2808683493 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30f00f66e0 0x7f30f010e3b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:25.214 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:25.226+0000 7f30f4d4c700 1 -- 172.21.15.139:0/2808683493 >> 172.21.15.139:0/2808683493 conn(0x7f30f00f0ba0 msgr2=0x7f30f00fa130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:25.214 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:25.227+0000 7f30f4d4c700 1 -- 172.21.15.139:0/2808683493 shutdown_connections 2021-11-20T06:03:25.214 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:25.227+0000 7f30f4d4c700 1 -- 172.21.15.139:0/2808683493 wait complete. 2021-11-20T06:03:25.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:25 smithi178 conmon[36600]: audit 2021-11-20T06:03:24.189130+0000 mon.smithi139 (mon.0) 445 : audit 2021-11-20T06:03:25.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:25 smithi178 conmon[36600]: [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2021-11-20T06:03:25.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:25 smithi178 conmon[36600]: cluster 2021-11-20T06:03:24.189176+0000 mon.smithi139 (mon.0) 446 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-11-20T06:03:25.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:25 smithi178 conmon[36600]: audit 2021-11-20T06:03:24.189988+0000 mon.smithi139 (mon. 2021-11-20T06:03:25.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:25 smithi178 conmon[36600]: 0) 447 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2021-11-20T06:03:25.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:25 smithi178 conmon[36600]: cluster 2021-11-20T06:03:24.207992+0000 mgr.smithi139.dplyxn 2021-11-20T06:03:25.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:25 smithi178 conmon[36600]: (mgr.14162) 183 : cluster [DBG] pgmap v145: 33 pgs: 32 unknown, 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:25.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:25 smithi139 conmon[33501]: audit 2021-11-20T06:03:24.189130+0000 mon.smithi139 (mon.0) 445 : audit 2021-11-20T06:03:25.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:25 smithi139 conmon[33501]: [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2021-11-20T06:03:25.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:25 smithi139 conmon[33501]: cluster 2021-11-20T06:03:24.189176+0000 mon.smithi139 (mon.0) 446 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-11-20T06:03:25.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:25 smithi139 conmon[33501]: audit 2021-11-20T06:03:24.189988+0000 mon.smithi139 (mon.0) 447 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2021-11-20T06:03:25.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:25 smithi139 conmon[33501]: cluster 2021-11- 2021-11-20T06:03:25.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:25 smithi139 conmon[33501]: 20T06:03:24.207992+0000 mgr.smithi139.dplyxn (mgr.14162) 183 : cluster [DBG] pgmap v145: 33 pgs: 32 unknown, 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:25.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:25 smithi139 conmon[33501]: 2021-11-20T06:03:25.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:25 smithi139 conmon[33501]: debug 2021-11-20T06:03:25.198+0000 7f2a1ee4f700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:03:25.607 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs dump' 2021-11-20T06:03:26.310 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: audit 2021-11-20T06:03:25.196538+0000 mon.smithi139 (mon.0) 448 2021-11-20T06:03:26.310 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2021-11-20T06:03:26.311 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: cluster 2021-11-20T06:03:25.196776+0000 mon.smithi139 (mon.0) 449 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-11-20T06:03:26.311 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: audit 2021-11-20T06:03:25.198563+0000 mon.smithi139 (mon.0) 450 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2021-11-20T06:03:26.312 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: cluster 2021-11-20T06:03:25.199427+0000 mon.smithi139 (mon.0) 451 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:03:26.312 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: cluster 2021-11-20T06:03:25.199464+0000 mon.smithi139 (mon.0) 452 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2021-11-20T06:03:26.313 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: cluster 2021-11-20T06:03:25.208302+0000 mon.smithi139 (mon.0) 453 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2021-11-20T06:03:26.313 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: audit 2021-11-20T06:03:25.208776+0000 mon.smithi139 (mon.0) 454 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2021-11-20T06:03:26.314 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: cluster 2021-11-20T06:03:25.208871+0000 mon.smithi139 (mon.0) 455 : cluster [DBG] fsmap cephfs:0 2021-11-20T06:03:26.314 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: cephadm 2021-11-20T06:03:25.212361+0000 mgr.smithi139.dplyxn (mgr.14162) 184 : cephadm [INF] Saving service mds.cephfs spec with placement count:4 2021-11-20T06:03:26.314 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: audit 2021-11-20T06:03:25.223430+0000 mon.smithi139 (mon.0) 456 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:26.315 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: audit 2021-11-20T06:03:25.227234+0000 mon.smithi139 (mon.0) 457 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:03:26.315 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: audit 2021-11-20T06:03:25.692896+0000 mon.smithi139 (mon.0) 458 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:26.315 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: audit 2021-11-20T06:03:25.699260+0000 mon.smithi139 (mon.0) 459 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:26.315 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: audit 2021-11-20T06:03:25.700040+0000 mon.smithi139 (mon.0) 460 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi178.blvkbx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T06:03:26.316 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: audit 2021-11-20T06:03:25.703295+0000 mon.smithi139 (mon.0) 461 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi178.blvkbx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T06:03:26.316 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: audit 2021-11-20T06:03:25.703899+0000 mon.smithi139 (mon.0) 462 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:03:26.316 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:26 smithi139 conmon[33501]: cephadm 2021-11-20T06:03:25.704437+0000 mgr.smithi139.dplyxn (mgr.14162) 185 : cephadm [INF] Deploying daemon mds.cephfs.smithi178.blvkbx on smithi178 2021-11-20T06:03:26.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: audit 2021-11-20T06:03:25.196538+0000 mon.smithi139 2021-11-20T06:03:26.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: (mon.0) 448 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2021-11-20T06:03:26.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: cluster 2021-11-20T06:03:25.196776+0000 mon.smithi139 (mon.0) 449 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-11-20T06:03:26.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: audit 2021-11-20T06:03:25.198563+0000 mon.smithi139 (mon.0) 450 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2021-11-20T06:03:26.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: cluster 2021- 2021-11-20T06:03:26.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 11-20T06:03:25.199427+0000 mon.smithi139 (mon.0) 451 2021-11-20T06:03:26.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:03:26.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: cluster 2021- 2021-11-20T06:03:26.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 11-20T06:03:25.199464+0000 mon.smithi139 ( 2021-11-20T06:03:26.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: mon.0) 452 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2021-11-20T06:03:26.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: cluster 2021-11-20T06:03:26.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 2021-11-20T06:03:25.208302+0000 mon.smithi139 2021-11-20T06:03:26.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: (mon.0) 453 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2021-11-20T06:03:26.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: audit 2021-11-20T06:03:26.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 2021-11-20T 2021-11-20T06:03:26.447 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 06:03:25.208776+0000 mon.smithi139 (mon.0 2021-11-20T06:03:26.447 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: ) 454 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2021-11-20T06:03:26.447 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: cluster 2021-11-20T06:03:26.447 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 2021-11-20T06:03: 2021-11-20T06:03:26.448 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 25.208871+0000 mon.smithi139 (mon.0) 455 : cluster 2021-11-20T06:03:26.448 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: [DBG] fsmap cephfs:0 2021-11-20T06:03:26.448 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: cephadm 2021-11-20T06:03:26.448 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 2021-11-20T06:03:25. 2021-11-20T06:03:26.448 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 212361+0000 mgr.smithi139.dplyxn (mgr.14162) 184 : cephadm 2021-11-20T06:03:26.449 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: [INF] Saving service mds.cephfs spec with placement count:4 2021-11-20T06:03:26.449 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: audit 2021-11-20T06 2021-11-20T06:03:26.449 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: :03:25.223430+0000 mon.smithi139 (mon.0 2021-11-20T06:03:26.449 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: ) 456 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:26.449 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: audit 2021-11-20T06: 2021-11-20T06:03:26.450 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 03:25.227234+0000 mon.smithi139 (mon.0) 2021-11-20T06:03:26.450 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 457 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:03:26.450 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: audit 2021-11-20T06:03:26.450 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 2021-11-20T06:03:25.692896+0000 mon.smithi139 (mon.0) 2021-11-20T06:03:26.451 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 458 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:26.451 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: audit 2021-11-20T06:03:26.451 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 2021-11-20T06:03:25.699260+0000 2021-11-20T06:03:26.451 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: mon.smithi139 (mon.0) 459 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:26.451 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 2021-11-20T06:03:26.452 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: audit 2021-11-20 2021-11-20T06:03:26.452 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: T06:03:25.700040+0000 mon.smithi139 (mon.0) 460 2021-11-20T06:03:26.452 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi178.blvkbx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T06:03:26.452 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: audit 2021-11-20T06:03:26.453 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 2021-11-20T06:03:25.703295+0000 2021-11-20T06:03:26.453 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: mon.smithi139 (mon.0) 461 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi178.blvkbx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T06:03:26.453 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 2021-11-20T06:03:26.453 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: audit 2021-11-20T 2021-11-20T06:03:26.453 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 06:03:25.703899+0000 mon.smithi139 (mon.0 2021-11-20T06:03:26.454 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: ) 462 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:03:26.454 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: cephadm 2021-11-20T06:03:26.454 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: 2021-11-20T06:03:25.704437+0000 mgr.smithi139.dplyxn 2021-11-20T06:03:26.454 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:26 smithi178 conmon[36600]: (mgr.14162) 185 : cephadm [INF] Deploying daemon mds.cephfs.smithi178.blvkbx on smithi178 2021-11-20T06:03:27.179 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.190+0000 7f05c36ae700 1 -- 172.21.15.139:0/3692455965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05bc0f7960 msgr2=0x7f05bc0f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:27.179 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.190+0000 7f05c36ae700 1 --2- 172.21.15.139:0/3692455965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05bc0f7960 0x7f05bc0f7dc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f05ac004660 tx=0x7f05ac009b30).stop 2021-11-20T06:03:27.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.190+0000 7f05c36ae700 1 -- 172.21.15.139:0/3692455965 shutdown_connections 2021-11-20T06:03:27.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.190+0000 7f05c36ae700 1 --2- 172.21.15.139:0/3692455965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05bc0f7960 0x7f05bc0f7dc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:27.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.190+0000 7f05c36ae700 1 --2- 172.21.15.139:0/3692455965 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05bc0f66c0 0x7f05bc0f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:27.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.190+0000 7f05c36ae700 1 -- 172.21.15.139:0/3692455965 >> 172.21.15.139:0/3692455965 conn(0x7f05bc0f1e20 msgr2=0x7f05bc0f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:27.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.190+0000 7f05c36ae700 1 -- 172.21.15.139:0/3692455965 shutdown_connections 2021-11-20T06:03:27.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.190+0000 7f05c36ae700 1 -- 172.21.15.139:0/3692455965 wait complete. 2021-11-20T06:03:27.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.191+0000 7f05c36ae700 1 Processor -- start 2021-11-20T06:03:27.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.191+0000 7f05c36ae700 1 -- start start 2021-11-20T06:03:27.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.191+0000 7f05c36ae700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05bc0f66c0 0x7f05bc10d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:27.183 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.191+0000 7f05c36ae700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05bc0f7960 0x7f05bc10d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:27.183 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.191+0000 7f05c36ae700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc10dfa0 con 0x7f05bc0f66c0 2021-11-20T06:03:27.184 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.191+0000 7f05c36ae700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc10e0e0 con 0x7f05bc0f7960 2021-11-20T06:03:27.184 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.191+0000 7f05c144a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05bc0f66c0 0x7f05bc10d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:27.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.191+0000 7f05c144a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05bc0f66c0 0x7f05bc10d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57380/0 (socket says 172.21.15.139:57380) 2021-11-20T06:03:27.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.191+0000 7f05c144a700 1 -- 172.21.15.139:0/1794351613 learned_addr learned my addr 172.21.15.139:0/1794351613 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:27.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.191+0000 7f05c0c49700 1 --2- 172.21.15.139:0/1794351613 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05bc0f7960 0x7f05bc10d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:27.186 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.191+0000 7f05c144a700 1 -- 172.21.15.139:0/1794351613 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05bc0f7960 msgr2=0x7f05bc10d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:27.186 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.191+0000 7f05c144a700 1 --2- 172.21.15.139:0/1794351613 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05bc0f7960 0x7f05bc10d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:27.187 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.191+0000 7f05c144a700 1 -- 172.21.15.139:0/1794351613 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac005040 con 0x7f05bc0f66c0 2021-11-20T06:03:27.187 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.191+0000 7f05c144a700 1 --2- 172.21.15.139:0/1794351613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05bc0f66c0 0x7f05bc10d500 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f05b800eb70 tx=0x7f05b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:27.187 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.192+0000 7f05b1ffb700 1 -- 172.21.15.139:0/1794351613 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05b80092e0 con 0x7f05bc0f66c0 2021-11-20T06:03:27.188 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.192+0000 7f05b1ffb700 1 -- 172.21.15.139:0/1794351613 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05b800f040 con 0x7f05bc0f66c0 2021-11-20T06:03:27.188 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.192+0000 7f05c36ae700 1 -- 172.21.15.139:0/1794351613 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc10ea80 con 0x7f05bc0f66c0 2021-11-20T06:03:27.189 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.192+0000 7f05b1ffb700 1 -- 172.21.15.139:0/1794351613 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05b800c8a0 con 0x7f05bc0f66c0 2021-11-20T06:03:27.189 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.192+0000 7f05c36ae700 1 -- 172.21.15.139:0/1794351613 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc10f040 con 0x7f05bc0f66c0 2021-11-20T06:03:27.189 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.193+0000 7f05c36ae700 1 -- 172.21.15.139:0/1794351613 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc045ad0 con 0x7f05bc0f66c0 2021-11-20T06:03:27.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.193+0000 7f05b1ffb700 1 -- 172.21.15.139:0/1794351613 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f05b801b030 con 0x7f05bc0f66c0 2021-11-20T06:03:27.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.193+0000 7f05b1ffb700 1 --2- 172.21.15.139:0/1794351613 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f05a805c2b0 0x7f05a805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:27.191 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.194+0000 7f05b1ffb700 1 -- 172.21.15.139:0/1794351613 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f05b8075ec0 con 0x7f05bc0f66c0 2021-11-20T06:03:27.191 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.194+0000 7f05c0c49700 1 --2- 172.21.15.139:0/1794351613 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f05a805c2b0 0x7f05a805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:27.191 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.194+0000 7f05c0c49700 1 --2- 172.21.15.139:0/1794351613 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f05a805c2b0 0x7f05a805e770 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f05ac005010 tx=0x7f05ac016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:27.192 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.198+0000 7f05b1ffb700 1 -- 172.21.15.139:0/1794351613 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05b8048c00 con 0x7f05bc0f66c0 2021-11-20T06:03:27.369 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.383+0000 7f05c36ae700 1 -- 172.21.15.139:0/1794351613 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f05bc0fefd0 con 0x7f05bc0f66c0 2021-11-20T06:03:27.369 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.384+0000 7f05b1ffb700 1 -- 172.21.15.139:0/1794351613 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1003 (secure 0 0 0) 0x7f05b804c220 con 0x7f05bc0f66c0 2021-11-20T06:03:27.370 INFO:teuthology.orchestra.run.smithi139.stdout:e2 2021-11-20T06:03:27.370 INFO:teuthology.orchestra.run.smithi139.stdout:enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T06:03:27.370 INFO:teuthology.orchestra.run.smithi139.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T06:03:27.371 INFO:teuthology.orchestra.run.smithi139.stdout:legacy client fscid: 1 2021-11-20T06:03:27.371 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:03:27.371 INFO:teuthology.orchestra.run.smithi139.stdout:Filesystem 'cephfs' (1) 2021-11-20T06:03:27.371 INFO:teuthology.orchestra.run.smithi139.stdout:fs_name cephfs 2021-11-20T06:03:27.372 INFO:teuthology.orchestra.run.smithi139.stdout:epoch 2 2021-11-20T06:03:27.372 INFO:teuthology.orchestra.run.smithi139.stdout:flags 12 2021-11-20T06:03:27.372 INFO:teuthology.orchestra.run.smithi139.stdout:created 2021-11-20T06:03:25.199256+0000 2021-11-20T06:03:27.372 INFO:teuthology.orchestra.run.smithi139.stdout:modified 2021-11-20T06:03:25.199330+0000 2021-11-20T06:03:27.372 INFO:teuthology.orchestra.run.smithi139.stdout:tableserver 0 2021-11-20T06:03:27.373 INFO:teuthology.orchestra.run.smithi139.stdout:root 0 2021-11-20T06:03:27.373 INFO:teuthology.orchestra.run.smithi139.stdout:session_timeout 60 2021-11-20T06:03:27.373 INFO:teuthology.orchestra.run.smithi139.stdout:session_autoclose 300 2021-11-20T06:03:27.373 INFO:teuthology.orchestra.run.smithi139.stdout:max_file_size 1099511627776 2021-11-20T06:03:27.373 INFO:teuthology.orchestra.run.smithi139.stdout:required_client_features {} 2021-11-20T06:03:27.374 INFO:teuthology.orchestra.run.smithi139.stdout:last_failure 0 2021-11-20T06:03:27.374 INFO:teuthology.orchestra.run.smithi139.stdout:last_failure_osd_epoch 0 2021-11-20T06:03:27.374 INFO:teuthology.orchestra.run.smithi139.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T06:03:27.374 INFO:teuthology.orchestra.run.smithi139.stdout:max_mds 1 2021-11-20T06:03:27.375 INFO:teuthology.orchestra.run.smithi139.stdout:in 2021-11-20T06:03:27.375 INFO:teuthology.orchestra.run.smithi139.stdout:up {} 2021-11-20T06:03:27.375 INFO:teuthology.orchestra.run.smithi139.stdout:failed 2021-11-20T06:03:27.375 INFO:teuthology.orchestra.run.smithi139.stdout:damaged 2021-11-20T06:03:27.375 INFO:teuthology.orchestra.run.smithi139.stdout:stopped 2021-11-20T06:03:27.376 INFO:teuthology.orchestra.run.smithi139.stdout:data_pools [3] 2021-11-20T06:03:27.376 INFO:teuthology.orchestra.run.smithi139.stdout:metadata_pool 2 2021-11-20T06:03:27.376 INFO:teuthology.orchestra.run.smithi139.stdout:inline_data disabled 2021-11-20T06:03:27.376 INFO:teuthology.orchestra.run.smithi139.stdout:balancer 2021-11-20T06:03:27.377 INFO:teuthology.orchestra.run.smithi139.stdout:standby_count_wanted 0 2021-11-20T06:03:27.377 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:03:27.377 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:03:27.378 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.385+0000 7f05c36ae700 1 -- 172.21.15.139:0/1794351613 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f05a805c2b0 msgr2=0x7f05a805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:27.378 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.385+0000 7f05c36ae700 1 --2- 172.21.15.139:0/1794351613 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f05a805c2b0 0x7f05a805e770 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f05ac005010 tx=0x7f05ac016040).stop 2021-11-20T06:03:27.378 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.385+0000 7f05c36ae700 1 -- 172.21.15.139:0/1794351613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05bc0f66c0 msgr2=0x7f05bc10d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:27.379 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.385+0000 7f05c36ae700 1 --2- 172.21.15.139:0/1794351613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05bc0f66c0 0x7f05bc10d500 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f05b800eb70 tx=0x7f05b800b040).stop 2021-11-20T06:03:27.379 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.386+0000 7f05c36ae700 1 -- 172.21.15.139:0/1794351613 shutdown_connections 2021-11-20T06:03:27.379 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.386+0000 7f05c36ae700 1 --2- 172.21.15.139:0/1794351613 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f05a805c2b0 0x7f05a805e770 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:27.379 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.386+0000 7f05c36ae700 1 --2- 172.21.15.139:0/1794351613 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05bc0f66c0 0x7f05bc10d500 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:27.380 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.386+0000 7f05c36ae700 1 --2- 172.21.15.139:0/1794351613 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05bc0f7960 0x7f05bc10d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:27.381 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.386+0000 7f05c36ae700 1 -- 172.21.15.139:0/1794351613 >> 172.21.15.139:0/1794351613 conn(0x7f05bc0f1e20 msgr2=0x7f05bc0f93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:27.382 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.386+0000 7f05c36ae700 1 -- 172.21.15.139:0/1794351613 shutdown_connections 2021-11-20T06:03:27.383 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:27.386+0000 7f05c36ae700 1 -- 172.21.15.139:0/1794351613 wait complete. 2021-11-20T06:03:27.383 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 2 2021-11-20T06:03:27.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:27 smithi139 conmon[33501]: cluster 2021-11-20T06:03:26.207972+0000 mon.smithi139 (mon.0) 463 : cluster [DBG] 2021-11-20T06:03:27.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:27 smithi139 conmon[33501]: osdmap e40: 6 total, 6 up, 6 in 2021-11-20T06:03:27.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:27 smithi139 conmon[33501]: cluster 2021-11-20T06:03:26.208310+0000 mgr.smithi139.dplyxn (mgr.14162) 186 : cluster [DBG] pgmap v149: 65 pgs: 17 creating+peering, 44 unknown, 4 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:27.539 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:27 smithi178 conmon[36600]: cluster 2021-11-20T06:03:26.207972+0000 mon.smithi139 (mon.0) 463 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2021-11-20T06:03:27.540 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:27 smithi178 conmon[36600]: cluster 2021-11-20T06:03:26.208310+0000 mgr.smithi139.dplyxn (mgr.14162) 186 : cluster [DBG] pgmap v149: 65 pgs: 17 creating+peering, 44 unknown, 4 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:27.825 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T06:03:27.838 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi139.front.sepia.ceph.com 2021-11-20T06:03:27.839 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs set cephfs max_mds 1' 2021-11-20T06:03:28.199 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:28 smithi139 conmon[33501]: cluster 2021-11-20T06:03:27.214390+0000 mon.smithi139 (mon.0 2021-11-20T06:03:28.199 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:28 smithi139 conmon[33501]: ) 464 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2021-11-20T06:03:28.199 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:28 smithi139 conmon[33501]: audit 2021-11-20T06:03:27.384611+0000 mon.smithi139 (mon.0) 465 : audit [DBG] from='client.? 172.21.15.139:0/1794351613' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T06:03:28.199 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:28 smithi139 conmon[33501]: audit 2021-11-20T06:03:27.888573+0000 mon.smithi139 (mon.0) 466 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:28.200 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:28 smithi139 conmon[33501]: audit 2021-11-20T06:03:27.889463+0000 mon.smithi139 (mon.0) 467 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi139.riczqb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T06:03:28.200 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:28 smithi139 conmon[33501]: audit 2021-11-20T06:03:27.892382+0000 mon.smithi139 (mon.0) 468 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi139.riczqb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T06:03:28.200 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:28 smithi139 conmon[33501]: audit 2021-11-20T06:03:27.893053+0000 mon.smithi139 (mon.0) 469 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:03:28.200 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:28 smithi139 conmon[33501]: cephadm 2021-11- 2021-11-20T06:03:28.201 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:28 smithi139 conmon[33501]: 20T06:03:27.893546+0000 mgr.smithi139.dplyxn (mgr 2021-11-20T06:03:28.201 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:28 smithi139 conmon[33501]: .14162) 187 : cephadm [INF] Deploying daemon mds.cephfs.smithi139.riczqb on smithi139 2021-11-20T06:03:28.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:28 smithi178 conmon[36600]: cluster 2021-11-20T06:03:27.214390+0000 mon.smithi139 (mon.0) 464 : 2021-11-20T06:03:28.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:28 smithi178 conmon[36600]: cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2021-11-20T06:03:28.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:28 smithi178 conmon[36600]: audit 2021-11-20T06:03:27.384611+0000 mon.smithi139 (mon.0) 465 : audit [DBG] from='client.? 172.21.15.139:0/1794351613' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T06:03:28.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:28 smithi178 conmon[36600]: audit 2021-11-20T06:03:27.888573+0000 mon.smithi139 (mon.0) 466 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:28.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:28 smithi178 conmon[36600]: audit 2021-11-20T06:03:27.889463+0000 mon.smithi139 (mon.0) 467 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi139.riczqb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T06:03:28.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:28 smithi178 conmon[36600]: audit 2021-11-20T06:03:27.892382+0000 mon.smithi139 (mon.0) 468 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi139.riczqb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T06:03:28.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:28 smithi178 conmon[36600]: audit 2021-11-20T06:03:27.893053+0000 mon.smithi139 (mon.0) 469 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:03:28.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:28 smithi178 conmon[36600]: cephadm 2021-11-20T06:03:27.893546+0000 mgr.smithi139.dplyxn (mgr.14162) 187 : cephadm [INF] Deploying daemon mds.cephfs.smithi139.riczqb on smithi139 2021-11-20T06:03:29.372 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:29 smithi139 conmon[33501]: cluster 2021-11-20T06:03:28.208673+0000 mgr.smithi139.dplyxn (mgr.14162) 188 : cluster [DBG] pgmap v151: 65 pgs: 17 creating+peering, 19 unknown, 29 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:29.372 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:29 smithi139 conmon[33501]: cluster 2021-11-20T06:03:28.221242+0000 mon.smithi139 (mon.0) 470 : cluster [DBG] mds.? [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] up:boot 2021-11-20T06:03:29.372 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:29 smithi139 conmon[33501]: cluster 2021-11-20T06:03:28.221302+0000 mon.smithi139 (mon.0) 471 : cluster [INF] daemon mds.cephfs.smithi178.blvkbx assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2021-11-20T06:03:29.373 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:29 smithi139 conmon[33501]: cluster 2021-11-20T06:03:28.221368+0000 mon.smithi139 (mon.0) 472 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:03:29.373 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:29 smithi139 conmon[33501]: cluster 2021-11-20T06:03:28.221392+0000 mon.smithi139 (mon.0) 473 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2021-11-20T06:03:29.373 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:29 smithi139 conmon[33501]: cluster 2021-11-20T06:03:28.221409+0000 mon.smithi139 (mon.0) 474 : cluster [INF] Cluster is now healthy 2021-11-20T06:03:29.373 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:29 smithi139 conmon[33501]: cluster 2021-11-20T06:03:28.221770+0000 mon.smithi139 (mon.0) 475 : cluster [DBG] fsmap cephfs:0 1 up:standby 2021-11-20T06:03:29.374 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:29 smithi139 conmon[33501]: audit 2021-11-20T06:03:28. 2021-11-20T06:03:29.374 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:29 smithi139 conmon[33501]: 221889+0000 mon.smithi139 (mon.0) 476 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.blvkbx"}]: dispatch 2021-11-20T06:03:29.374 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:29 smithi139 conmon[33501]: cluster 2021-11-20T06:03:28.226872+0000 mon.smithi139 (mon.0) 477 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:creating} 2021-11-20T06:03:29.374 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:29 smithi139 conmon[33501]: cluster 2021-11-20T06:03:28.237284+0000 mon.smithi139 (mon.0) 478 : cluster [INF] daemon mds.cephfs.smithi178.blvkbx is now active in filesystem cephfs as rank 0 2021-11-20T06:03:29.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: cluster 2021-11-20T06:03:28.208673+0000 mgr.smithi139.dplyxn (mgr.14162) 188 : cluster [DBG] pgmap v151: 65 pgs: 17 creating+peering, 19 unknown, 29 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:29.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: 2021-11-20T06:03:29.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: cluster 2021-11-20T06:03:28.221242+0000 mon.smithi139 (mon.0) 470 : cluster [DBG] mds.? [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] up:boot 2021-11-20T06:03:29.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: cluster 2021-11-20T06:03:28.221302+0000 2021-11-20T06:03:29.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: mon.smithi139 (mon.0) 471 : cluster [INF] daemon mds.cephfs.smithi178.blvkbx assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2021-11-20T06:03:29.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: cluster 2021-11-20T06:03:29.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: 2021-11-20T06:03: 2021-11-20T06:03:29.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: 28.221368+0000 mon.smithi139 (mon.0) 472 : 2021-11-20T06:03:29.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:03:29.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: cluster 2021-11 2021-11-20T06:03:29.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: -20T06:03:28.221392+0000 2021-11-20T06:03:29.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: mon.smithi139 (mon.0) 473 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2021-11-20T06:03:29.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: cluster 2021 2021-11-20T06:03:29.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: -11-20T06:03:28 2021-11-20T06:03:29.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: .221409+0000 mon.smithi139 (mon.0) 474 : cluster 2021-11-20T06:03:29.697 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: [INF] Cluster is now healthy 2021-11-20T06:03:29.697 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: cluster 2021-11- 2021-11-20T06:03:29.697 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: 20T06:03:28.221770+0000 2021-11-20T06:03:29.698 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: mon.smithi139 (mon.0) 475 : cluster [DBG] fsmap cephfs:0 1 up:standby 2021-11-20T06:03:29.698 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: 2021-11-20T06:03:29.698 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: audit 2021-11-20T06 2021-11-20T06:03:29.698 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: :03:28.221889+0000 mon.smithi139 (mon. 2021-11-20T06:03:29.698 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: 0) 476 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.blvkbx"}]: dispatch 2021-11-20T06:03:29.699 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: cluster 2021 2021-11-20T06:03:29.699 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: -11-20T06:03:28. 2021-11-20T06:03:29.699 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: 226872+0000 mon.smithi139 (mon.0) 477 : cluster 2021-11-20T06:03:29.699 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:creating} 2021-11-20T06:03:29.700 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: cluster 2021-11- 2021-11-20T06:03:29.700 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: 20T06:03:28.237284+0000 2021-11-20T06:03:29.700 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: mon.smithi139 (mon.0) 478 : cluster [INF] daemon mds.cephfs.smithi178.blvkbx is now active in filesystem cephfs as rank 0 2021-11-20T06:03:29.700 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:29 smithi178 conmon[36600]: 2021-11-20T06:03:29.793 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.806+0000 7f16bdb55700 1 -- 172.21.15.139:0/987010444 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16b80f7980 msgr2=0x7f16b80f7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:29.793 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.806+0000 7f16bdb55700 1 --2- 172.21.15.139:0/987010444 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16b80f7980 0x7f16b80f7de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f16a0004660 tx=0x7f16a0009b30).stop 2021-11-20T06:03:29.794 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.806+0000 7f16bdb55700 1 -- 172.21.15.139:0/987010444 shutdown_connections 2021-11-20T06:03:29.794 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.806+0000 7f16bdb55700 1 --2- 172.21.15.139:0/987010444 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16b80f7980 0x7f16b80f7de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:29.794 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.806+0000 7f16bdb55700 1 --2- 172.21.15.139:0/987010444 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16b80f66e0 0x7f16b80f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:29.794 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.806+0000 7f16bdb55700 1 -- 172.21.15.139:0/987010444 >> 172.21.15.139:0/987010444 conn(0x7f16b80f1e40 msgr2=0x7f16b80f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:29.795 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.806+0000 7f16bdb55700 1 -- 172.21.15.139:0/987010444 shutdown_connections 2021-11-20T06:03:29.795 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.806+0000 7f16bdb55700 1 -- 172.21.15.139:0/987010444 wait complete. 2021-11-20T06:03:29.795 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.806+0000 7f16bdb55700 1 Processor -- start 2021-11-20T06:03:29.795 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.806+0000 7f16bdb55700 1 -- start start 2021-11-20T06:03:29.796 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.806+0000 7f16bdb55700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16b80f66e0 0x7f16b806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:29.796 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.806+0000 7f16bdb55700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16b80f7980 0x7f16b8069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:29.796 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.806+0000 7f16bdb55700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b806a380 con 0x7f16b80f66e0 2021-11-20T06:03:29.796 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.806+0000 7f16bdb55700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b806a4c0 con 0x7f16b80f7980 2021-11-20T06:03:29.797 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.807+0000 7f16b77fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16b80f66e0 0x7f16b806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:29.797 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.807+0000 7f16b77fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16b80f66e0 0x7f16b806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57390/0 (socket says 172.21.15.139:57390) 2021-11-20T06:03:29.797 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.807+0000 7f16b77fe700 1 -- 172.21.15.139:0/80041351 learned_addr learned my addr 172.21.15.139:0/80041351 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:29.797 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.807+0000 7f16b6ffd700 1 --2- 172.21.15.139:0/80041351 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16b80f7980 0x7f16b8069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:29.797 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.807+0000 7f16b77fe700 1 -- 172.21.15.139:0/80041351 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16b80f7980 msgr2=0x7f16b8069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:29.798 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.807+0000 7f16b77fe700 1 --2- 172.21.15.139:0/80041351 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16b80f7980 0x7f16b8069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:29.798 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.807+0000 7f16b77fe700 1 -- 172.21.15.139:0/80041351 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a0005040 con 0x7f16b80f66e0 2021-11-20T06:03:29.798 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.807+0000 7f16b77fe700 1 --2- 172.21.15.139:0/80041351 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16b80f66e0 0x7f16b806b9a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f16b000eb70 tx=0x7f16b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:29.798 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.807+0000 7f16affff700 1 -- 172.21.15.139:0/80041351 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16b00092e0 con 0x7f16b80f66e0 2021-11-20T06:03:29.799 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.807+0000 7f16bdb55700 1 -- 172.21.15.139:0/80041351 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b806a7a0 con 0x7f16b80f66e0 2021-11-20T06:03:29.799 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.807+0000 7f16affff700 1 -- 172.21.15.139:0/80041351 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16b000f040 con 0x7f16b80f66e0 2021-11-20T06:03:29.799 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.807+0000 7f16affff700 1 -- 172.21.15.139:0/80041351 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16b000c8a0 con 0x7f16b80f66e0 2021-11-20T06:03:29.799 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.807+0000 7f16bdb55700 1 -- 172.21.15.139:0/80041351 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b810ef30 con 0x7f16b80f66e0 2021-11-20T06:03:29.801 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.808+0000 7f16bdb55700 1 -- 172.21.15.139:0/80041351 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b8045ad0 con 0x7f16b80f66e0 2021-11-20T06:03:29.801 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.809+0000 7f16affff700 1 -- 172.21.15.139:0/80041351 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f16b001b030 con 0x7f16b80f66e0 2021-11-20T06:03:29.801 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.809+0000 7f16affff700 1 --2- 172.21.15.139:0/80041351 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f16a405c2b0 0x7f16a405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:29.801 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.809+0000 7f16affff700 1 -- 172.21.15.139:0/80041351 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f16b0075f90 con 0x7f16b80f66e0 2021-11-20T06:03:29.802 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.810+0000 7f16b6ffd700 1 --2- 172.21.15.139:0/80041351 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f16a405c2b0 0x7f16a405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:29.802 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.810+0000 7f16b6ffd700 1 --2- 172.21.15.139:0/80041351 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f16a405c2b0 0x7f16a405e770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f16a0005010 tx=0x7f16a0016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:29.802 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:29.814+0000 7f16affff700 1 -- 172.21.15.139:0/80041351 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16b0048c50 con 0x7f16b80f66e0 2021-11-20T06:03:30.011 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:30.025+0000 7f16bdb55700 1 -- 172.21.15.139:0/80041351 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f16b80fefb0 con 0x7f16b80f66e0 2021-11-20T06:03:30.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:30.230+0000 7f16affff700 1 -- 172.21.15.139:0/80041351 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f16b004c270 con 0x7f16b80f66e0 2021-11-20T06:03:30.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:30.233+0000 7f16bdb55700 1 -- 172.21.15.139:0/80041351 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f16a405c2b0 msgr2=0x7f16a405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:30.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:30.233+0000 7f16bdb55700 1 --2- 172.21.15.139:0/80041351 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f16a405c2b0 0x7f16a405e770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f16a0005010 tx=0x7f16a0016040).stop 2021-11-20T06:03:30.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:30.233+0000 7f16bdb55700 1 -- 172.21.15.139:0/80041351 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16b80f66e0 msgr2=0x7f16b806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:30.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:30.233+0000 7f16bdb55700 1 --2- 172.21.15.139:0/80041351 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16b80f66e0 0x7f16b806b9a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f16b000eb70 tx=0x7f16b000b040).stop 2021-11-20T06:03:30.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:30.234+0000 7f16bdb55700 1 -- 172.21.15.139:0/80041351 shutdown_connections 2021-11-20T06:03:30.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:30.234+0000 7f16bdb55700 1 --2- 172.21.15.139:0/80041351 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f16a405c2b0 0x7f16a405e770 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:30.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:30.234+0000 7f16bdb55700 1 --2- 172.21.15.139:0/80041351 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f16b80f66e0 0x7f16b806b9a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:30.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:30.234+0000 7f16bdb55700 1 --2- 172.21.15.139:0/80041351 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16b80f7980 0x7f16b8069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:30.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:30.234+0000 7f16bdb55700 1 -- 172.21.15.139:0/80041351 >> 172.21.15.139:0/80041351 conn(0x7f16b80f1e40 msgr2=0x7f16b80fb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:30.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:30.234+0000 7f16bdb55700 1 -- 172.21.15.139:0/80041351 shutdown_connections 2021-11-20T06:03:30.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:30.234+0000 7f16bdb55700 1 -- 172.21.15.139:0/80041351 wait complete. 2021-11-20T06:03:30.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:30 smithi139 conmon[33501]: cluster 2021-11-20T06:03:29.226793+0000 mon.smithi139 (mon.0) 479 : cluster [DBG] mds.? [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] up:active 2021-11-20T06:03:30.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:30 smithi139 conmon[33501]: cluster 2021-11-20T06:03 2021-11-20T06:03:30.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:30 smithi139 conmon[33501]: :29.226856+0000 mon.smithi139 (mon.0) 480 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 2021-11-20T06:03:30.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:30 smithi139 conmon[33501]: audit 2021-11-20T06:03:30.026434+0000 mon.smithi139 (mon.0) 481 : audit [INF] from='client.? 172.21.15.139:0/80041351' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2021-11-20T06:03:30.555 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T06:03:30.567 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi139.front.sepia.ceph.com 2021-11-20T06:03:30.568 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2021-11-20T06:03:30.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:30 smithi178 conmon[36600]: cluster 2021-11-20T06:03:29.226793+0000 mon.smithi139 (mon.0) 479 : cluster [DBG] mds.? [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] up:active 2021-11-20T06:03:30.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:30 smithi178 conmon[36600]: cluster 2021-11-20T06:03:29.226856+0000 mon.smithi139 (mon.0) 2021-11-20T06:03:30.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:30 smithi178 conmon[36600]: 480 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 2021-11-20T06:03:30.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:30 smithi178 conmon[36600]: audit 2021-11-20T06:03:30.026434+0000 mon.smithi139 (mon.0) 481 : audit [INF] from='client.? 172.21.15.139:0/80041351' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2021-11-20T06:03:31.391 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:31 smithi139 conmon[33501]: cluster 2021-11-20T06:03:30.209053+0000 2021-11-20T06:03:31.391 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:31 smithi139 conmon[33501]: mgr.smithi139.dplyxn (mgr.14162) 189 : cluster [DBG] pgmap v152: 65 pgs: 3 creating+peering, 62 active+clean; 2.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:31.391 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:31 smithi139 conmon[33501]: audit 2021-11-20T06:03:30.231489+0000 mon.smithi139 (mon.0) 482 : audit [INF] from='client.? 172.21.15.139:0/80041351' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2021-11-20T06:03:31.392 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:31 smithi139 conmon[33501]: cluster 2021-11-20T06:03:30.231564+0000 mon.smithi139 (mon.0) 483 : cluster [DBG] mds.? [v2:172.21.15.139:6826/724620583,v1:172.21.15.139:6827/724620583] up:boot 2021-11-20T06:03:31.392 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:31 smithi139 conmon[33501]: cluster 2021-11-20T06:03:30.231815+0000 mon.smithi139 (mon.0) 484 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby 2021-11-20T06:03:31.392 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:31 smithi139 conmon[33501]: audit 2021-11-20T06:03:30.231913+0000 mon.smithi139 (mon.0) 485 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.riczqb"}]: dispatch 2021-11-20T06:03:31.392 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:31 smithi139 conmon[33501]: cluster 2021-11-20T06:03:30.234678+0000 mon.smithi139 (mon.0) 486 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby 2021-11-20T06:03:31.393 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:31 smithi139 conmon[33501]: audit 2021-11-20T06:03:30.289685+0000 mon.smithi139 (mon.0) 487 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:31.393 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:31 smithi139 conmon[33501]: audit 2021-11-20T06:03:30.290435+0000 mon.smithi139 (mon.0) 488 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi178.pskihy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T06:03:31.393 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:31 smithi139 conmon[33501]: audit 2021-11-20T06:03:30.293176+0000 mon.smithi139 (mon.0) 489 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi178.pskihy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T06:03:31.393 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:31 smithi139 conmon[33501]: audit 2021-11-20T06:03:30.293793+0000 mon.smithi139 (mon.0) 490 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:03:31.394 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:31 smithi139 conmon[33501]: cephadm 2021-11-20T06:03:30.294294+0000 mgr.smithi139.dplyxn (mgr.14162) 190 : cephadm [INF] Deploying daemon mds.cephfs.smithi178.pskihy on smithi178 2021-11-20T06:03:31.486 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: cluster 2021-11-20T06:03:30.209053+0000 mgr.smithi139.dplyxn (mgr.14162) 189 : cluster [DBG] pgmap v152: 65 pgs: 3 creating+peering, 62 active+clean; 2.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:03:31.486 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: audit 2021-11-20T06:03:30.231489+0000 mon.smithi139 (mon.0) 482 : audit [INF] from='client.? 172.21.15.139:0/80041351' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2021-11-20T06:03:31.486 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: cluster 2021-11-20T06:03:30.231564+0000 mon.smithi139 (mon.0) 483 : cluster [DBG] mds.? [v2:172.21.15.139:6826/724620583,v1:172.21.15.139:6827/724620583] up:boot 2021-11-20T06:03:31.487 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: cluster 2021-11-20T06:03:30.231815+0000 mon.smithi139 (mon.0) 484 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby 2021-11-20T06:03:31.487 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: audit 2021-11-20T06:03:30.231913+0000 mon.smithi139 (mon.0) 485 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.riczqb"}]: dispatch 2021-11-20T06:03:31.487 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: cluster 2021-11-20T06:03:30 2021-11-20T06:03:31.488 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: .234678+0000 mon.smithi139 (mon.0) 486 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby 2021-11-20T06:03:31.488 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: audit 2021-11-20T06:03:30 2021-11-20T06:03:31.488 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: .289685+0000 mon.smithi139 (mon.0) 487 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:31.488 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: audit 2021-11-20T06:03:31.488 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: 2021-11-20T06:03: 2021-11-20T06:03:31.489 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: 30.290435+0000 mon.smithi139 (mon.0) 488 : 2021-11-20T06:03:31.489 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi178.pskihy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T06:03:31.489 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: audit 2021-11-20 2021-11-20T06:03:31.489 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: T06:03:30.293176+0000 mon.smithi139 2021-11-20T06:03:31.490 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: (mon.0) 489 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi178.pskihy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T06:03:31.490 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: audit 2021-11-20T06:03:31.490 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: 2021-11-20T06:03 2021-11-20T06:03:31.490 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: :30.293793+0000 mon.smithi139 (mon.0) 490 2021-11-20T06:03:31.490 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:03:31.491 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: cephadm 2021-11 2021-11-20T06:03:31.491 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: -20T06:03:30.294294 2021-11-20T06:03:31.491 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: +0000 mgr.smithi139.dplyxn (mgr.14162) 190 : cephadm [INF] 2021-11-20T06:03:31.491 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:31 smithi178 conmon[36600]: Deploying daemon mds.cephfs.smithi178.pskihy on smithi178 2021-11-20T06:03:32.008 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.021+0000 7f9099b34700 1 -- 172.21.15.139:0/1272852093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90940f55e0 msgr2=0x7f90940f5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:32.008 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.021+0000 7f9099b34700 1 --2- 172.21.15.139:0/1272852093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90940f55e0 0x7f90940f5a00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f9084004660 tx=0x7f9084009b30).stop 2021-11-20T06:03:32.008 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.021+0000 7f9099b34700 1 -- 172.21.15.139:0/1272852093 shutdown_connections 2021-11-20T06:03:32.009 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.021+0000 7f9099b34700 1 --2- 172.21.15.139:0/1272852093 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90940f6880 0x7f90940f6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:32.009 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.021+0000 7f9099b34700 1 --2- 172.21.15.139:0/1272852093 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90940f55e0 0x7f90940f5a00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:32.010 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.021+0000 7f9099b34700 1 -- 172.21.15.139:0/1272852093 >> 172.21.15.139:0/1272852093 conn(0x7f90940f0d40 msgr2=0x7f90940f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:32.010 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.021+0000 7f9099b34700 1 -- 172.21.15.139:0/1272852093 shutdown_connections 2021-11-20T06:03:32.010 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.021+0000 7f9099b34700 1 -- 172.21.15.139:0/1272852093 wait complete. 2021-11-20T06:03:32.011 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.021+0000 7f9099b34700 1 Processor -- start 2021-11-20T06:03:32.011 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.022+0000 7f9099b34700 1 -- start start 2021-11-20T06:03:32.011 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.022+0000 7f9099b34700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90940f55e0 0x7f9094105ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:32.011 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.022+0000 7f9099b34700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90940f6880 0x7f90941063b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:32.012 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.022+0000 7f9099b34700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90941069b0 con 0x7f90940f6880 2021-11-20T06:03:32.012 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.022+0000 7f9099b34700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90941071c0 con 0x7f90940f55e0 2021-11-20T06:03:32.012 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.022+0000 7f9092ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90940f6880 0x7f90941063b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:32.012 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.022+0000 7f9092ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90940f6880 0x7f90941063b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57410/0 (socket says 172.21.15.139:57410) 2021-11-20T06:03:32.013 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.022+0000 7f90937fe700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90940f55e0 0x7f9094105ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:32.013 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.022+0000 7f9092ffd700 1 -- 172.21.15.139:0/2800867291 learned_addr learned my addr 172.21.15.139:0/2800867291 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:32.013 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.022+0000 7f90937fe700 1 -- 172.21.15.139:0/2800867291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90940f6880 msgr2=0x7f90941063b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:32.013 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.022+0000 7f90937fe700 1 --2- 172.21.15.139:0/2800867291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90940f6880 0x7f90941063b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:32.014 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.022+0000 7f90937fe700 1 -- 172.21.15.139:0/2800867291 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9084005040 con 0x7f90940f55e0 2021-11-20T06:03:32.014 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.022+0000 7f90937fe700 1 --2- 172.21.15.139:0/2800867291 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90940f55e0 0x7f9094105ec0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9084004660 tx=0x7f908400b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:32.014 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.022+0000 7f907bfff700 1 -- 172.21.15.139:0/2800867291 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9084019070 con 0x7f90940f55e0 2021-11-20T06:03:32.014 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.022+0000 7f9099b34700 1 -- 172.21.15.139:0/2800867291 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90941073e0 con 0x7f90940f55e0 2021-11-20T06:03:32.014 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.023+0000 7f907bfff700 1 -- 172.21.15.139:0/2800867291 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9084007e70 con 0x7f90940f55e0 2021-11-20T06:03:32.015 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.023+0000 7f907bfff700 1 -- 172.21.15.139:0/2800867291 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9084014cd0 con 0x7f90940f55e0 2021-11-20T06:03:32.015 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.023+0000 7f9099b34700 1 -- 172.21.15.139:0/2800867291 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9094107940 con 0x7f90940f55e0 2021-11-20T06:03:32.016 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.023+0000 7f9099b34700 1 -- 172.21.15.139:0/2800867291 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90940f49d0 con 0x7f90940f55e0 2021-11-20T06:03:32.016 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.024+0000 7f907bfff700 1 -- 172.21.15.139:0/2800867291 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9084025070 con 0x7f90940f55e0 2021-11-20T06:03:32.017 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.024+0000 7f907bfff700 1 --2- 172.21.15.139:0/2800867291 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f907c05c2a0 0x7f907c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:32.017 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.024+0000 7f907bfff700 1 -- 172.21.15.139:0/2800867291 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f9084076de0 con 0x7f90940f55e0 2021-11-20T06:03:32.017 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.024+0000 7f9092ffd700 1 --2- 172.21.15.139:0/2800867291 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f907c05c2a0 0x7f907c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:32.017 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.025+0000 7f9092ffd700 1 --2- 172.21.15.139:0/2800867291 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f907c05c2a0 0x7f907c05e760 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9088004cf0 tx=0x7f908800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:32.017 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.029+0000 7f907bfff700 1 -- 172.21.15.139:0/2800867291 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f90840499e0 con 0x7f90940f55e0 2021-11-20T06:03:32.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:32.220+0000 7f9099b34700 1 -- 172.21.15.139:0/2800867291 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f90940fdf00 con 0x7f90940f55e0 2021-11-20T06:03:32.277 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:32 smithi139 conmon[33501]: audit 2021-11-20T06:03:31.252179+0000 mon.smithi139 ( 2021-11-20T06:03:32.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:32 smithi139 conmon[33501]: mon.0) 491 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:03:32.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:32 smithi139 conmon[33501]: audit 2021-11-20T06:03:31.252407+0000 mon.smithi139 (mon.0) 492 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:03:32.278 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:32 smithi139 conmon[33501]: audit 2021-11-20T06:03:32.221860+0000 mon.smithi178 (mon.1) 17 : audit [INF] from='client.? 172.21.15.139:0/2800867291' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2021-11-20T06:03:32.279 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:32 smithi139 conmon[33501]: audit 2021-11-20T06:03:32.222128+0000 mon.smithi139 (mon.0) 493 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2021-11-20T06:03:32.393 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:32 smithi178 conmon[36600]: audit 2021-11-20T06:03:31.252179+0000 mon.smithi139 (mon.0) 491 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:03:32.394 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:32 smithi178 conmon[36600]: audit 2021-11 2021-11-20T06:03:32.394 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:32 smithi178 conmon[36600]: -20T06:03:31.252407+0000 mon.smithi139 (mon.0) 492 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:03:32.394 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:32 smithi178 conmon[36600]: audit 2021-11-20T06:03:32.221860+0000 mon.smithi178 (mon.1) 17 : audit [INF] from='client.? 172.21.15.139:0/2800867291' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2021-11-20T06:03:32.395 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:32 smithi178 conmon[36600]: audit 2021-11-20T06:03:32.222128+0000 mon.smithi139 (mon.0) 493 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2021-11-20T06:03:33.173 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:33.187+0000 7f907bfff700 1 -- 172.21.15.139:0/2800867291 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v8) v1 ==== 121+0+0 (secure 0 0 0) 0x7f90940fdf00 con 0x7f90940f55e0 2021-11-20T06:03:33.176 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:33.189+0000 7f9099b34700 1 -- 172.21.15.139:0/2800867291 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f907c05c2a0 msgr2=0x7f907c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:33.176 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:33.189+0000 7f9099b34700 1 --2- 172.21.15.139:0/2800867291 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f907c05c2a0 0x7f907c05e760 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9088004cf0 tx=0x7f908800b040).stop 2021-11-20T06:03:33.176 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:33.189+0000 7f9099b34700 1 -- 172.21.15.139:0/2800867291 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90940f55e0 msgr2=0x7f9094105ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:33.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:33.189+0000 7f9099b34700 1 --2- 172.21.15.139:0/2800867291 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90940f55e0 0x7f9094105ec0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9084004660 tx=0x7f908400b040).stop 2021-11-20T06:03:33.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:33.189+0000 7f9099b34700 1 -- 172.21.15.139:0/2800867291 shutdown_connections 2021-11-20T06:03:33.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:33.189+0000 7f9099b34700 1 --2- 172.21.15.139:0/2800867291 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f907c05c2a0 0x7f907c05e760 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:33.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:33.189+0000 7f9099b34700 1 --2- 172.21.15.139:0/2800867291 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90940f55e0 0x7f9094105ec0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:33.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:33.189+0000 7f9099b34700 1 --2- 172.21.15.139:0/2800867291 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90940f6880 0x7f90941063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:33.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:33.189+0000 7f9099b34700 1 -- 172.21.15.139:0/2800867291 >> 172.21.15.139:0/2800867291 conn(0x7f90940f0d40 msgr2=0x7f90940fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:33.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:33.190+0000 7f9099b34700 1 -- 172.21.15.139:0/2800867291 shutdown_connections 2021-11-20T06:03:33.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:33.190+0000 7f9099b34700 1 -- 172.21.15.139:0/2800867291 wait complete. 2021-11-20T06:03:33.587 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-20T06:03:33.608 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi139.front.sepia.ceph.com 2021-11-20T06:03:33.609 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs dump' 2021-11-20T06:03:33.634 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: cluster 2021-11-20T06:03:32.209422+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:03:33.635 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: mgr.14162) 191 : cluster [DBG] pgmap v153: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s wr, 4 op/s 2021-11-20T06:03:33.635 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: audit 2021-11-20T06:03:32.479203+0000 mon.smithi139 (mon.0) 494 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:33.635 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: audit 2021-11-20T06:03:32.479902+0000 mon.smithi139 (mon.0) 495 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi139.oudezh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T06:03:33.635 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: audit 2021-11-20T06:03:32.484996+0000 mon.smithi139 (mon.0) 496 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi139.oudezh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T06:03:33.635 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: audit 2021-11-20T06:03:32.485676+0000 mon.smithi139 (mon.0) 497 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:03:33.636 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: cephadm 2021-11-20T 2021-11-20T06:03:33.636 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: 06:03:32.486172+0000 mgr.smithi139.dplyxn (mgr.14162) 192 : cephadm [INF] Deploying daemon mds.cephfs.smithi139.oudezh on smithi139 2021-11-20T06:03:33.636 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: audit 2021-11-20T06:03:33.187682+0000 mon.smithi139 ( 2021-11-20T06:03:33.636 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: mon.0) 498 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2021-11-20T06:03:33.637 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: cluster 2021-11-20T06: 2021-11-20T06:03:33.637 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: 03:33.187735+0000 mon.smithi139 (mon.0 2021-11-20T06:03:33.637 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: ) 499 : cluster [DBG] mds.? [v2:172.21.15.178:6826/614777783,v1:172.21.15.178:6827/614777783] up:boot 2021-11-20T06:03:33.637 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: cluster 2021-11-20 2021-11-20T06:03:33.638 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: T06:03:33.188428+0000 mon.smithi139 ( 2021-11-20T06:03:33.638 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: mon.0) 500 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 2 up:standby 2021-11-20T06:03:33.638 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: audit 2021- 2021-11-20T06:03:33.639 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: 11-20T06:03:33.188525+0000 2021-11-20T06:03:33.639 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: mon.smithi139 (mon.0) 501 : audit [DBG] 2021-11-20T06:03:33.639 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.pskihy"}]: dispatch 2021-11-20T06:03:33.639 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: cluster 2021-11-20T06:03:33.640 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: 2021-11-20T06:03:33.193229+0000 mon.smithi139 2021-11-20T06:03:33.640 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:33 smithi139 conmon[33501]: (mon.0) 502 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby-replay 1 up:standby 2021-11-20T06:03:33.942 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: cluster 2021-11-20T06:03:32.209422+0000 mgr.smithi139.dplyxn (mgr.14162) 191 : cluster [DBG] pgmap v153: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s wr, 4 op/s 2021-11-20T06:03:33.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: audit 2021-11-20T06:03:32.479203+0000 mon.smithi139 (mon.0) 494 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:33.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: audit 2021-11-20T06:03:32.479902 2021-11-20T06:03:33.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: +0000 mon.smithi139 (mon.0) 495 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi139.oudezh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-20T06:03:33.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: audit 2021-11-20T06:03:33.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: 2021-11-20T06:03:32.484996+0000 mon.smithi139 (mon.0) 2021-11-20T06:03:33.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: 496 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi139.oudezh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-20T06:03:33.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: audit 2021-11-20T06:03: 2021-11-20T06:03:33.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: 32.485676+0000 mon.smithi139 (mon.0) 497 : audit 2021-11-20T06:03:33.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-20T06:03:33.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: cephadm 2021 2021-11-20T06:03:33.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: -11-20T06:03:32.486172+0000 2021-11-20T06:03:33.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: mgr.smithi139.dplyxn (mgr.14162) 192 : cephadm [INF] Deploying daemon mds.cephfs.smithi139.oudezh on smithi139 2021-11-20T06:03:33.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: audit 2021-11- 2021-11-20T06:03:33.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: 20T06:03:33.187682+0000 mon.smithi139 ( 2021-11-20T06:03:33.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: mon.0) 498 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2021-11-20T06:03:33.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: cluster 2021- 2021-11-20T06:03:33.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: 11-20T06:03:33.187735+0000 2021-11-20T06:03:33.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: mon.smithi139 (mon.0) 499 : cluster [DBG] mds.? [v2:172.21.15.178:6826/614777783,v1:172.21.15.178:6827/614777783] up:boot 2021-11-20T06:03:33.948 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: 2021-11-20T06:03:33.948 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: cluster 2021- 2021-11-20T06:03:33.949 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: 11-20T06:03:33.188428+0000 mon.smithi139 (mon.0) 500 : 2021-11-20T06:03:33.949 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 2 up:standby 2021-11-20T06:03:33.949 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: audit 2021 2021-11-20T06:03:33.949 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: -11-20T06:03:33.188525+0000 mon.smithi139 (mon.0) 501 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.pskihy"}]: dispatch 2021-11-20T06:03:33.950 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: cluster 2021-11-20T06:03: 2021-11-20T06:03:33.950 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:33 smithi178 conmon[36600]: 33.193229+0000 mon.smithi139 (mon.0) 502 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby-replay 1 up:standby 2021-11-20T06:03:35.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.253+0000 7f1b55169700 1 -- 172.21.15.139:0/4067043924 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1b500f9b80 msgr2=0x7f1b500f9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:35.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.253+0000 7f1b55169700 1 --2- 172.21.15.139:0/4067043924 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1b500f9b80 0x7f1b500f9fe0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f1b44004660 tx=0x7f1b44009b30).stop 2021-11-20T06:03:35.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.253+0000 7f1b55169700 1 -- 172.21.15.139:0/4067043924 shutdown_connections 2021-11-20T06:03:35.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.253+0000 7f1b55169700 1 --2- 172.21.15.139:0/4067043924 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1b500f9b80 0x7f1b500f9fe0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:35.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.253+0000 7f1b55169700 1 --2- 172.21.15.139:0/4067043924 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b500f88e0 0x7f1b500f8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:03:35.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.253+0000 7f1b55169700 1 -- 172.21.15.139:0/4067043924 >> 172.21.15.139:0/4067043924 conn(0x7f1b500f40e0 msgr2=0x7f1b500f64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:35.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.253+0000 7f1b55169700 1 -- 172.21.15.139:0/4067043924 shutdown_connections 2021-11-20T06:03:35.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.253+0000 7f1b55169700 1 -- 172.21.15.139:0/4067043924 wait complete. 2021-11-20T06:03:35.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.253+0000 7f1b55169700 1 Processor -- start 2021-11-20T06:03:35.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b55169700 1 -- start start 2021-11-20T06:03:35.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b55169700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1b500f88e0 0x7f1b501091a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:35.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b55169700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b500f9b80 0x7f1b50109690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:35.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b55169700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b50109cd0 con 0x7f1b500f88e0 2021-11-20T06:03:35.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b55169700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5010a580 con 0x7f1b500f9b80 2021-11-20T06:03:35.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b4ed9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1b500f88e0 0x7f1b501091a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:35.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b4ed9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1b500f88e0 0x7f1b501091a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57434/0 (socket says 172.21.15.139:57434) 2021-11-20T06:03:35.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b4ed9d700 1 -- 172.21.15.139:0/662619531 learned_addr learned my addr 172.21.15.139:0/662619531 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:35.246 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b4e59c700 1 --2- 172.21.15.139:0/662619531 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b500f9b80 0x7f1b50109690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:35.246 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b4ed9d700 1 -- 172.21.15.139:0/662619531 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b500f9b80 msgr2=0x7f1b50109690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:35.246 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b4ed9d700 1 --2- 172.21.15.139:0/662619531 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b500f9b80 0x7f1b50109690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:35.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b4ed9d700 1 -- 172.21.15.139:0/662619531 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b44005040 con 0x7f1b500f88e0 2021-11-20T06:03:35.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b4ed9d700 1 --2- 172.21.15.139:0/662619531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1b500f88e0 0x7f1b501091a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1b400093f0 tx=0x7f1b40006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:35.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b377fe700 1 -- 172.21.15.139:0/662619531 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b40013640 con 0x7f1b500f88e0 2021-11-20T06:03:35.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b377fe700 1 -- 172.21.15.139:0/662619531 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b40013c50 con 0x7f1b500f88e0 2021-11-20T06:03:35.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b55169700 1 -- 172.21.15.139:0/662619531 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5010a860 con 0x7f1b500f88e0 2021-11-20T06:03:35.248 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b377fe700 1 -- 172.21.15.139:0/662619531 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b40012c80 con 0x7f1b500f88e0 2021-11-20T06:03:35.248 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.254+0000 7f1b55169700 1 -- 172.21.15.139:0/662619531 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5010ae20 con 0x7f1b500f88e0 2021-11-20T06:03:35.249 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.256+0000 7f1b377fe700 1 -- 172.21.15.139:0/662619531 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1b4001a060 con 0x7f1b500f88e0 2021-11-20T06:03:35.249 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.256+0000 7f1b55169700 1 -- 172.21.15.139:0/662619531 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b500f7cd0 con 0x7f1b500f88e0 2021-11-20T06:03:35.250 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.256+0000 7f1b377fe700 1 --2- 172.21.15.139:0/662619531 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1b3805c2b0 0x7f1b3805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:35.250 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.256+0000 7f1b377fe700 1 -- 172.21.15.139:0/662619531 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f1b40075ef0 con 0x7f1b500f88e0 2021-11-20T06:03:35.250 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.256+0000 7f1b4e59c700 1 --2- 172.21.15.139:0/662619531 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1b3805c2b0 0x7f1b3805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:35.250 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.257+0000 7f1b4e59c700 1 --2- 172.21.15.139:0/662619531 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1b3805c2b0 0x7f1b3805e770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f1b44004630 tx=0x7f1b44009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:35.251 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.260+0000 7f1b377fe700 1 -- 172.21.15.139:0/662619531 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b40048bb0 con 0x7f1b500f88e0 2021-11-20T06:03:35.441 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.455+0000 7f1b55169700 1 -- 172.21.15.139:0/662619531 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1b50027a00 con 0x7f1b500f88e0 2021-11-20T06:03:35.441 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.455+0000 7f1b377fe700 1 -- 172.21.15.139:0/662619531 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 11 v11) v1 ==== 76+0+1313 (secure 0 0 0) 0x7f1b4004c1d0 con 0x7f1b500f88e0 2021-11-20T06:03:35.442 INFO:teuthology.orchestra.run.smithi139.stdout:e11 2021-11-20T06:03:35.442 INFO:teuthology.orchestra.run.smithi139.stdout:enable_multiple, ever_enabled_multiple: 1,1 2021-11-20T06:03:35.443 INFO:teuthology.orchestra.run.smithi139.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T06:03:35.443 INFO:teuthology.orchestra.run.smithi139.stdout:legacy client fscid: 1 2021-11-20T06:03:35.443 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:03:35.444 INFO:teuthology.orchestra.run.smithi139.stdout:Filesystem 'cephfs' (1) 2021-11-20T06:03:35.444 INFO:teuthology.orchestra.run.smithi139.stdout:fs_name cephfs 2021-11-20T06:03:35.444 INFO:teuthology.orchestra.run.smithi139.stdout:epoch 11 2021-11-20T06:03:35.444 INFO:teuthology.orchestra.run.smithi139.stdout:flags 32 2021-11-20T06:03:35.445 INFO:teuthology.orchestra.run.smithi139.stdout:created 2021-11-20T06:03:25.199256+0000 2021-11-20T06:03:35.445 INFO:teuthology.orchestra.run.smithi139.stdout:modified 2021-11-20T06:03:34.486673+0000 2021-11-20T06:03:35.446 INFO:teuthology.orchestra.run.smithi139.stdout:tableserver 0 2021-11-20T06:03:35.446 INFO:teuthology.orchestra.run.smithi139.stdout:root 0 2021-11-20T06:03:35.446 INFO:teuthology.orchestra.run.smithi139.stdout:session_timeout 60 2021-11-20T06:03:35.447 INFO:teuthology.orchestra.run.smithi139.stdout:session_autoclose 300 2021-11-20T06:03:35.447 INFO:teuthology.orchestra.run.smithi139.stdout:max_file_size 1099511627776 2021-11-20T06:03:35.448 INFO:teuthology.orchestra.run.smithi139.stdout:required_client_features {} 2021-11-20T06:03:35.448 INFO:teuthology.orchestra.run.smithi139.stdout:last_failure 0 2021-11-20T06:03:35.448 INFO:teuthology.orchestra.run.smithi139.stdout:last_failure_osd_epoch 0 2021-11-20T06:03:35.448 INFO:teuthology.orchestra.run.smithi139.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-20T06:03:35.449 INFO:teuthology.orchestra.run.smithi139.stdout:max_mds 1 2021-11-20T06:03:35.449 INFO:teuthology.orchestra.run.smithi139.stdout:in 0 2021-11-20T06:03:35.449 INFO:teuthology.orchestra.run.smithi139.stdout:up {0=24237} 2021-11-20T06:03:35.450 INFO:teuthology.orchestra.run.smithi139.stdout:failed 2021-11-20T06:03:35.450 INFO:teuthology.orchestra.run.smithi139.stdout:damaged 2021-11-20T06:03:35.451 INFO:teuthology.orchestra.run.smithi139.stdout:stopped 2021-11-20T06:03:35.451 INFO:teuthology.orchestra.run.smithi139.stdout:data_pools [3] 2021-11-20T06:03:35.451 INFO:teuthology.orchestra.run.smithi139.stdout:metadata_pool 2 2021-11-20T06:03:35.452 INFO:teuthology.orchestra.run.smithi139.stdout:inline_data disabled 2021-11-20T06:03:35.452 INFO:teuthology.orchestra.run.smithi139.stdout:balancer 2021-11-20T06:03:35.453 INFO:teuthology.orchestra.run.smithi139.stdout:standby_count_wanted 1 2021-11-20T06:03:35.453 INFO:teuthology.orchestra.run.smithi139.stdout:[mds.cephfs.smithi178.blvkbx{0:24237} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351]] 2021-11-20T06:03:35.453 INFO:teuthology.orchestra.run.smithi139.stdout:[mds.cephfs.smithi139.riczqb{0:14426} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.139:6826/724620583,v1:172.21.15.139:6827/724620583]] 2021-11-20T06:03:35.454 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:03:35.454 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:03:35.455 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.458+0000 7f1b55169700 1 -- 172.21.15.139:0/662619531 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1b3805c2b0 msgr2=0x7f1b3805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:35.455 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.458+0000 7f1b55169700 1 --2- 172.21.15.139:0/662619531 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1b3805c2b0 0x7f1b3805e770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f1b44004630 tx=0x7f1b44009b30).stop 2021-11-20T06:03:35.455 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.458+0000 7f1b55169700 1 -- 172.21.15.139:0/662619531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1b500f88e0 msgr2=0x7f1b501091a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:35.456 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.458+0000 7f1b55169700 1 --2- 172.21.15.139:0/662619531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1b500f88e0 0x7f1b501091a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1b400093f0 tx=0x7f1b40006ec0).stop 2021-11-20T06:03:35.456 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.458+0000 7f1b55169700 1 -- 172.21.15.139:0/662619531 shutdown_connections 2021-11-20T06:03:35.456 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.458+0000 7f1b55169700 1 --2- 172.21.15.139:0/662619531 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1b3805c2b0 0x7f1b3805e770 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:35.456 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.458+0000 7f1b55169700 1 --2- 172.21.15.139:0/662619531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1b500f88e0 0x7f1b501091a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:35.456 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.458+0000 7f1b55169700 1 --2- 172.21.15.139:0/662619531 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b500f9b80 0x7f1b50109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:35.457 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.458+0000 7f1b55169700 1 -- 172.21.15.139:0/662619531 >> 172.21.15.139:0/662619531 conn(0x7f1b500f40e0 msgr2=0x7f1b500fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:35.457 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.458+0000 7f1b55169700 1 -- 172.21.15.139:0/662619531 shutdown_connections 2021-11-20T06:03:35.457 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:35.458+0000 7f1b55169700 1 -- 172.21.15.139:0/662619531 wait complete. 2021-11-20T06:03:35.457 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 11 2021-11-20T06:03:35.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:35 smithi139 conmon[33501]: cluster 2021-11-20T06:03:34.209793+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:03:35.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:35 smithi139 conmon[33501]: ) 193 : cluster [DBG] pgmap v154: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 767 B/s wr, 2 op/s 2021-11-20T06:03:35.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:35 smithi139 conmon[33501]: cluster 2021-11-20T06:03:34.486601+0000 mon.smithi139 (mon.0) 503 : cluster [DBG] mds.? [v2:172.21.15.139:6826/724620583,v1:172.21.15.139:6827/724620583] up:standby 2021-11-20T06:03:35.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:35 smithi139 conmon[33501]: cluster 2021-11-20T06:03:34.486636+0000 mon.smithi139 (mon.0) 504 : cluster [INF] Dropping low affinity standby-replay daemon mds.cephfs.smithi178.pskihy in favor of higher affinity standby. 2021-11-20T06:03:35.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:35 smithi139 conmon[33501]: cluster 2021-11-20T06:03:34.486893+0000 mon.smithi139 (mon.0) 505 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby-replay 1 up:standby 2021-11-20T06:03:35.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:35 smithi139 conmon[33501]: cluster 2021-11-20T06:03:34.488773+0000 mon.smithi139 (mon.0) 506 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby-replay 2021-11-20T06:03:35.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:35 smithi139 conmon[33501]: audit 2021-11-20T06:03:35.039572+0000 mon.smithi139 (mon.0) 507 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:35.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:35 smithi139 conmon[33501]: audit 2021-11-20T06:03:35.040243+0000 mon.smithi139 (mon.0) 508 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:03:35.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:35 smithi139 conmon[33501]: audit 2021-11-20T06:03:35.456361+0000 mon.smithi139 (mon.0) 509 : audit [DBG] from='client.? 172.21.15.139:0/662619531' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T06:03:35.842 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2021-11-20T06:03:35.942 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:35 smithi178 conmon[36600]: cluster 2021-11-20T06:03:34.209793+0000 mgr.smithi139.dplyxn (mgr.14162) 193 : cluster [DBG] pgmap v154: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 767 B/s wr, 2 op/s 2021-11-20T06:03:35.942 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:35 smithi178 conmon[36600]: cluster 2021-11-20T06:03:34.486601+0000 mon.smithi139 (mon.0) 503 : cluster [DBG] mds.? [v2:172.21.15.139:6826/724620583,v1:172.21.15.139:6827/724620583] up:standby 2021-11-20T06:03:35.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:35 smithi178 conmon[36600]: cluster 2021-11-20T06:03:34.486636+0000 mon.smithi139 (mon.0) 504 : cluster [INF] Dropping low affinity standby-replay daemon mds.cephfs.smithi178.pskihy in favor of higher affinity standby. 2021-11-20T06:03:35.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:35 smithi178 conmon[36600]: cluster 2021-11-20T06:03:34.486893+0000 mon.smithi139 (mon.0) 505 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby-replay 1 up:standby 2021-11-20T06:03:35.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:35 smithi178 conmon[36600]: cluster 2021-11-20T06:03:34.488773+0000 mon.smithi139 (mon.0) 506 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby-replay 2021-11-20T06:03:35.943 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:35 smithi178 conmon[36600]: audit 2021-11-20T06:03:35.039572+0000 mon.smithi139 (mon.0) 507 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:35.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:35 smithi178 conmon[36600]: audit 2021-11-20T06:03:35.040243+0000 mon.smithi139 (mon. 2021-11-20T06:03:35.944 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:35 smithi178 conmon[36600]: 0) 508 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:03:35.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:35 smithi178 conmon[36600]: audit 2021-11-20T06:03:35.456361+0000 mon.smithi139 (mon.0) 509 : audit [DBG] from='client.? 172.21.15.139:0/662619531' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-20T06:03:36.681 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:36 smithi139 conmon[33501]: cluster 2021-11-20T06:03:35.494993+0000 mon.smithi139 (mon.0) 510 : cluster [DBG] mds.? [v2:172.21.15.178:6826/1921979486,v1:172.21.15.178:6827/1921979486] up:boot 2021-11-20T06:03:36.682 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:36 smithi139 conmon[33501]: cluster 2021-11-20T06:03:35.495066+0000 mon.smithi139 (mon.0) 511 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby-replay 1 up:standby 2021-11-20T06:03:36.682 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:36 smithi139 conmon[33501]: audit 2021-11-20T06:03:35.495195+0000 mon.smithi139 (mon.0) 512 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.pskihy"}]: dispatch 2021-11-20T06:03:36.945 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:36 smithi178 conmon[36600]: cluster 2021-11-20T06:03:35.494993+0000 2021-11-20T06:03:36.946 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:36 smithi178 conmon[36600]: mon.smithi139 (mon.0) 510 : cluster [DBG] mds.? [v2:172.21.15.178:6826/1921979486,v1:172.21.15.178:6827/1921979486] up:boot 2021-11-20T06:03:36.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:36 smithi178 conmon[36600]: cluster 2021-11-20T06:03:35.495066+0000 mon.smithi139 (mon.0) 511 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby-replay 1 up:standby 2021-11-20T06:03:36.947 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:36 smithi178 conmon[36600]: audit 2021-11-20T06:03:35.495195+0000 mon.smithi139 (mon.0) 512 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.pskihy"}]: dispatch 2021-11-20T06:03:37.848 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.861+0000 7f887897d700 1 -- 172.21.15.139:0/1490458178 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88740f66e0 msgr2=0x7f88740f6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:37.849 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.861+0000 7f887897d700 1 --2- 172.21.15.139:0/1490458178 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88740f66e0 0x7f88740f6b40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f885c004660 tx=0x7f885c009b30).stop 2021-11-20T06:03:37.849 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.861+0000 7f887897d700 1 -- 172.21.15.139:0/1490458178 shutdown_connections 2021-11-20T06:03:37.849 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.861+0000 7f887897d700 1 --2- 172.21.15.139:0/1490458178 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88740f66e0 0x7f88740f6b40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:37.850 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.861+0000 7f887897d700 1 --2- 172.21.15.139:0/1490458178 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88740f5440 0x7f88740f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:37.850 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.861+0000 7f887897d700 1 -- 172.21.15.139:0/1490458178 >> 172.21.15.139:0/1490458178 conn(0x7f88740f0ba0 msgr2=0x7f88740f2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:37.850 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.861+0000 7f887897d700 1 -- 172.21.15.139:0/1490458178 shutdown_connections 2021-11-20T06:03:37.850 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.861+0000 7f887897d700 1 -- 172.21.15.139:0/1490458178 wait complete. 2021-11-20T06:03:37.851 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.862+0000 7f887897d700 1 Processor -- start 2021-11-20T06:03:37.851 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.862+0000 7f887897d700 1 -- start start 2021-11-20T06:03:37.851 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.862+0000 7f887897d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88740f5440 0x7f88741078a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:37.851 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.862+0000 7f887897d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88740f66e0 0x7f8874109da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:37.852 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.862+0000 7f887897d700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88740603e0 con 0x7f88740f66e0 2021-11-20T06:03:37.852 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.862+0000 7f887897d700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8874060520 con 0x7f88740f5440 2021-11-20T06:03:37.852 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.862+0000 7f886a59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88740f66e0 0x7f8874109da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:37.852 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.862+0000 7f886a59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88740f66e0 0x7f8874109da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57444/0 (socket says 172.21.15.139:57444) 2021-11-20T06:03:37.853 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.862+0000 7f886a59c700 1 -- 172.21.15.139:0/2732201037 learned_addr learned my addr 172.21.15.139:0/2732201037 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:37.853 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.862+0000 7f8872d9d700 1 --2- 172.21.15.139:0/2732201037 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88740f5440 0x7f88741078a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:37.853 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.862+0000 7f886a59c700 1 -- 172.21.15.139:0/2732201037 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88740f5440 msgr2=0x7f88741078a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:37.853 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.862+0000 7f886a59c700 1 --2- 172.21.15.139:0/2732201037 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88740f5440 0x7f88741078a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:37.854 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.862+0000 7f886a59c700 1 -- 172.21.15.139:0/2732201037 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885c005040 con 0x7f88740f66e0 2021-11-20T06:03:37.854 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.862+0000 7f886a59c700 1 --2- 172.21.15.139:0/2732201037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88740f66e0 0x7f8874109da0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f885c009f50 tx=0x7f885c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:37.854 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.863+0000 7f886bfff700 1 -- 172.21.15.139:0/2732201037 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f885c018070 con 0x7f88740f66e0 2021-11-20T06:03:37.854 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.863+0000 7f886bfff700 1 -- 172.21.15.139:0/2732201037 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f885c013410 con 0x7f88740f66e0 2021-11-20T06:03:37.855 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.863+0000 7f887897d700 1 -- 172.21.15.139:0/2732201037 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f887410a2f0 con 0x7f88740f66e0 2021-11-20T06:03:37.855 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.863+0000 7f886bfff700 1 -- 172.21.15.139:0/2732201037 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f885c01c3c0 con 0x7f88740f66e0 2021-11-20T06:03:37.855 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.863+0000 7f887897d700 1 -- 172.21.15.139:0/2732201037 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887410a850 con 0x7f88740f66e0 2021-11-20T06:03:37.855 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.863+0000 7f887897d700 1 -- 172.21.15.139:0/2732201037 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8874059100 con 0x7f88740f66e0 2021-11-20T06:03:37.856 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.864+0000 7f886bfff700 1 -- 172.21.15.139:0/2732201037 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f885c01d420 con 0x7f88740f66e0 2021-11-20T06:03:37.857 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.864+0000 7f886bfff700 1 --2- 172.21.15.139:0/2732201037 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f886005c350 0x7f886005e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:37.857 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.865+0000 7f886bfff700 1 -- 172.21.15.139:0/2732201037 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f885c077c00 con 0x7f88740f66e0 2021-11-20T06:03:37.857 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.865+0000 7f8872d9d700 1 --2- 172.21.15.139:0/2732201037 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f886005c350 0x7f886005e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:37.857 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.865+0000 7f8872d9d700 1 --2- 172.21.15.139:0/2732201037 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f886005c350 0x7f886005e810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f8874060060 tx=0x7f886400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:37.858 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:37.868+0000 7f886bfff700 1 -- 172.21.15.139:0/2732201037 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f885c04a800 con 0x7f88740f66e0 2021-11-20T06:03:38.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:38.076+0000 7f887897d700 1 -- 172.21.15.139:0/2732201037 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88740fdbd0 con 0x7f88740f66e0 2021-11-20T06:03:38.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:38.076+0000 7f886bfff700 1 -- 172.21.15.139:0/2732201037 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2568 (secure 0 0 0) 0x7f885c04de20 con 0x7f88740f66e0 2021-11-20T06:03:38.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:38.078+0000 7f887897d700 1 -- 172.21.15.139:0/2732201037 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f886005c350 msgr2=0x7f886005e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:38.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:38.078+0000 7f887897d700 1 --2- 172.21.15.139:0/2732201037 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f886005c350 0x7f886005e810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f8874060060 tx=0x7f886400b040).stop 2021-11-20T06:03:38.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:38.078+0000 7f887897d700 1 -- 172.21.15.139:0/2732201037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88740f66e0 msgr2=0x7f8874109da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:38.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:38.078+0000 7f887897d700 1 --2- 172.21.15.139:0/2732201037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88740f66e0 0x7f8874109da0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f885c009f50 tx=0x7f885c009b30).stop 2021-11-20T06:03:38.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:38.078+0000 7f887897d700 1 -- 172.21.15.139:0/2732201037 shutdown_connections 2021-11-20T06:03:38.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:38.078+0000 7f887897d700 1 --2- 172.21.15.139:0/2732201037 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f886005c350 0x7f886005e810 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:38.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:38.078+0000 7f887897d700 1 --2- 172.21.15.139:0/2732201037 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88740f5440 0x7f88741078a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:38.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:38.078+0000 7f887897d700 1 --2- 172.21.15.139:0/2732201037 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f88740f66e0 0x7f8874109da0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:38.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:38.078+0000 7f887897d700 1 -- 172.21.15.139:0/2732201037 >> 172.21.15.139:0/2732201037 conn(0x7f88740f0ba0 msgr2=0x7f88740fa130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:38.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:38.079+0000 7f887897d700 1 -- 172.21.15.139:0/2732201037 shutdown_connections 2021-11-20T06:03:38.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:38.079+0000 7f887897d700 1 -- 172.21.15.139:0/2732201037 wait complete. 2021-11-20T06:03:38.068 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 12 2021-11-20T06:03:38.075 INFO:teuthology.orchestra.run.smithi139.stdout:true 2021-11-20T06:03:38.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:38 smithi178 conmon[36600]: cluster 2021-11-20T06:03:36.210192+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:03:38.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:38 smithi178 conmon[36600]: 14162) 194 : cluster [DBG] pgmap v155: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2021-11-20T06:03:38.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:38 smithi178 conmon[36600]: audit 2021-11-20T06:03:38.016514+0000 mon.smithi139 (mon.0) 513 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:38.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:38 smithi178 conmon[36600]: audit 2021-11-20T06:03:38.077363+0000 mon.smithi139 (mon.0) 514 : audit [DBG] from='client.? 172.21.15.139:0/2732201037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:03:38.449 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:38 smithi139 conmon[33501]: cluster 2021-11-20T06:03:36.210192+0000 mgr.smithi139.dplyxn (mgr.14162) 194 : cluster [DBG] pgmap v155: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2021-11-20T06:03:38.449 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:38 smithi139 conmon[33501]: audit 2021-11-20T06:03:38.016514+0000 mon.smithi139 (mon.0) 513 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:38.450 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:38 smithi139 conmon[33501]: audit 2021-11-20T06:03:38.077363+0000 mon.smithi139 (mon.0) 514 : audit [DBG] from='client.? 172.21.15.139:0/2732201037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:03:38.603 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c -- bash -c 'ceph --format=json mds versions | jq -e ". | add == 4"' 2021-11-20T06:03:40.192 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:39 smithi178 conmon[36600]: cluster 2021-11-20T06:03:38.210744+0000 mgr.smithi139.dplyxn (mgr.14162) 195 : cluster [DBG] pgmap v156: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 383 B/s wr, 2 op/s 2021-11-20T06:03:40.192 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:39 smithi178 conmon[36600]: cluster 2021-11-20T06:03:39.186595+0000 mon.smithi139 (mon.0) 515 : cluster [DBG] mds.? [v2:172.21.15.178:6826/1921979486,v1:172.21.15.178:6827/1921979486] up:standby 2021-11-20T06:03:40.192 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:39 smithi178 conmon[36600]: cluster 2021-11-20T06:03:39.186624+0000 mon.smithi139 (mon.0) 516 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2967963544,v1:172.21.15.139:6829/2967963544] up:boot 2021-11-20T06:03:40.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:39 smithi178 conmon[36600]: cluster 2021-11-20T06:03:39.186674+0000 mon.smithi139 (mon.0) 517 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby-replay 2 up:standby 2021-11-20T06:03:40.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:39 smithi178 conmon[36600]: audit 2021-11-20T06:03:39.186757+0000 mon.smithi139 (mon.0) 518 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.oudezh"}]: dispatch 2021-11-20T06:03:40.229 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.242+0000 7f619ef28700 1 -- 172.21.15.139:0/660108885 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61980f5420 msgr2=0x7f61980fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:40.229 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.242+0000 7f619ef28700 1 --2- 172.21.15.139:0/660108885 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61980f5420 0x7f61980fbd30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f6184004660 tx=0x7f6184009b30).stop 2021-11-20T06:03:40.229 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.243+0000 7f619ef28700 1 -- 172.21.15.139:0/660108885 shutdown_connections 2021-11-20T06:03:40.230 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.243+0000 7f619ef28700 1 --2- 172.21.15.139:0/660108885 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61980f5420 0x7f61980fbd30 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:40.230 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.243+0000 7f619ef28700 1 --2- 172.21.15.139:0/660108885 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61980f4a80 0x7f61980f4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:40.230 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.243+0000 7f619ef28700 1 -- 172.21.15.139:0/660108885 >> 172.21.15.139:0/660108885 conn(0x7f61980f04c0 msgr2=0x7f61980f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:40.230 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.243+0000 7f619ef28700 1 -- 172.21.15.139:0/660108885 shutdown_connections 2021-11-20T06:03:40.231 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.243+0000 7f619ef28700 1 -- 172.21.15.139:0/660108885 wait complete. 2021-11-20T06:03:40.231 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:39 smithi139 conmon[33501]: cluster 2021-11-20T06:03:38.210744+0000 mgr.smithi139.dplyxn (mgr.14162) 195 : cluster [DBG] pgmap v156: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 383 B/s wr, 2 op/s 2021-11-20T06:03:40.231 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:39 smithi139 conmon[33501]: cluster 2021-11-20T06:03:39.186595+0000 mon.smithi139 (mon.0) 515 : cluster [DBG] mds.? [v2:172.21.15.178:6826/1921979486,v1:172.21.15.178:6827/1921979486] up:standby 2021-11-20T06:03:40.232 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:39 smithi139 conmon[33501]: cluster 2021-11-20T06:03:39.186624+0000 mon.smithi139 (mon.0) 516 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2967963544,v1:172.21.15.139:6829/2967963544] up:boot 2021-11-20T06:03:40.232 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:39 smithi139 conmon[33501]: cluster 2021-11-20T06:03:39.186674+0000 mon.smithi139 (mon.0) 517 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby-replay 2 up:standby 2021-11-20T06:03:40.232 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:39 smithi139 conmon[33501]: audit 2021-11-20T06:03:39.186757+0000 mon.smithi139 (mon.0) 518 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.oudezh"}]: dispatch 2021-11-20T06:03:40.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.243+0000 7f619ef28700 1 Processor -- start 2021-11-20T06:03:40.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.243+0000 7f619ef28700 1 -- start start 2021-11-20T06:03:40.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f619ef28700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61980f4a80 0x7f6198106fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:40.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f619ef28700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61980f5420 0x7f61981074b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:40.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f619ef28700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6198107a60 con 0x7f61980f5420 2021-11-20T06:03:40.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f619ef28700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6198107ba0 con 0x7f61980f4a80 2021-11-20T06:03:40.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f6197fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61980f5420 0x7f61981074b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:40.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f6197fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61980f5420 0x7f61981074b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57454/0 (socket says 172.21.15.139:57454) 2021-11-20T06:03:40.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f619ccc4700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61980f4a80 0x7f6198106fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:40.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f6197fff700 1 -- 172.21.15.139:0/478928574 learned_addr learned my addr 172.21.15.139:0/478928574 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:40.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f6197fff700 1 -- 172.21.15.139:0/478928574 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61980f4a80 msgr2=0x7f6198106fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:40.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f6197fff700 1 --2- 172.21.15.139:0/478928574 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61980f4a80 0x7f6198106fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:40.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f6197fff700 1 -- 172.21.15.139:0/478928574 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6184005040 con 0x7f61980f5420 2021-11-20T06:03:40.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f6197fff700 1 --2- 172.21.15.139:0/478928574 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61980f5420 0x7f61981074b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f6184004630 tx=0x7f6184009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:40.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f61957fa700 1 -- 172.21.15.139:0/478928574 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6184018070 con 0x7f61980f5420 2021-11-20T06:03:40.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f619ef28700 1 -- 172.21.15.139:0/478928574 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61981084e0 con 0x7f61980f5420 2021-11-20T06:03:40.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f61957fa700 1 -- 172.21.15.139:0/478928574 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6184007c70 con 0x7f61980f5420 2021-11-20T06:03:40.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f61957fa700 1 -- 172.21.15.139:0/478928574 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f618401c620 con 0x7f61980f5420 2021-11-20T06:03:40.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.244+0000 7f619ef28700 1 -- 172.21.15.139:0/478928574 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6198108a40 con 0x7f61980f5420 2021-11-20T06:03:40.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.246+0000 7f619ef28700 1 -- 172.21.15.139:0/478928574 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6198045ad0 con 0x7f61980f5420 2021-11-20T06:03:40.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.246+0000 7f61957fa700 1 -- 172.21.15.139:0/478928574 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f618401c780 con 0x7f61980f5420 2021-11-20T06:03:40.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.246+0000 7f61957fa700 1 --2- 172.21.15.139:0/478928574 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f618805c2b0 0x7f618805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:40.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.246+0000 7f61957fa700 1 -- 172.21.15.139:0/478928574 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f6184077b50 con 0x7f61980f5420 2021-11-20T06:03:40.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.246+0000 7f619ccc4700 1 --2- 172.21.15.139:0/478928574 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f618805c2b0 0x7f618805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:40.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.246+0000 7f619ccc4700 1 --2- 172.21.15.139:0/478928574 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f618805c2b0 0x7f618805e770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f619805edd0 tx=0x7f618c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:40.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.250+0000 7f61957fa700 1 -- 172.21.15.139:0/478928574 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f618404a890 con 0x7f61980f5420 2021-11-20T06:03:40.449 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.463+0000 7f619ef28700 1 -- 172.21.15.139:0/478928574 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f6198027a00 con 0x7f61980f5420 2021-11-20T06:03:40.449 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.463+0000 7f61957fa700 1 -- 172.21.15.139:0/478928574 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v13) v1 ==== 78+0+86 (secure 0 0 0) 0x7f6184004490 con 0x7f61980f5420 2021-11-20T06:03:40.451 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.465+0000 7f619ef28700 1 -- 172.21.15.139:0/478928574 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f618805c2b0 msgr2=0x7f618805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:40.452 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.465+0000 7f619ef28700 1 --2- 172.21.15.139:0/478928574 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f618805c2b0 0x7f618805e770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f619805edd0 tx=0x7f618c009040).stop 2021-11-20T06:03:40.452 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.465+0000 7f619ef28700 1 -- 172.21.15.139:0/478928574 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61980f5420 msgr2=0x7f61981074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:40.453 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.465+0000 7f619ef28700 1 --2- 172.21.15.139:0/478928574 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61980f5420 0x7f61981074b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f6184004630 tx=0x7f6184009b30).stop 2021-11-20T06:03:40.453 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.465+0000 7f619ef28700 1 -- 172.21.15.139:0/478928574 shutdown_connections 2021-11-20T06:03:40.453 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.465+0000 7f619ef28700 1 --2- 172.21.15.139:0/478928574 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f618805c2b0 0x7f618805e770 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:40.454 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.465+0000 7f619ef28700 1 --2- 172.21.15.139:0/478928574 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61980f4a80 0x7f6198106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:40.454 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.465+0000 7f619ef28700 1 --2- 172.21.15.139:0/478928574 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61980f5420 0x7f61981074b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:40.454 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.466+0000 7f619ef28700 1 -- 172.21.15.139:0/478928574 >> 172.21.15.139:0/478928574 conn(0x7f61980f04c0 msgr2=0x7f61980f8530 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:40.456 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.466+0000 7f619ef28700 1 -- 172.21.15.139:0/478928574 shutdown_connections 2021-11-20T06:03:40.456 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:40.466+0000 7f619ef28700 1 -- 172.21.15.139:0/478928574 wait complete. 2021-11-20T06:03:40.462 INFO:teuthology.orchestra.run.smithi139.stdout:true 2021-11-20T06:03:40.825 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2021-11-20T06:03:40.875 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-11-20T06:03:41.035 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.047+0000 7f38b6911700 1 -- 172.21.15.139:0/92830481 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38b00f6c20 msgr2=0x7f38b00f7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:41.035 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.047+0000 7f38b6911700 1 --2- 172.21.15.139:0/92830481 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38b00f6c20 0x7f38b00f7040 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f38a4004660 tx=0x7f38a4009b30).stop 2021-11-20T06:03:41.035 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.047+0000 7f38b6911700 1 -- 172.21.15.139:0/92830481 shutdown_connections 2021-11-20T06:03:41.036 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.047+0000 7f38b6911700 1 --2- 172.21.15.139:0/92830481 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38b00f5930 0x7f38b00f5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.036 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.047+0000 7f38b6911700 1 --2- 172.21.15.139:0/92830481 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38b00f6c20 0x7f38b00f7040 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.036 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.047+0000 7f38b6911700 1 -- 172.21.15.139:0/92830481 >> 172.21.15.139:0/92830481 conn(0x7f38b00f0440 msgr2=0x7f38b00f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:41.036 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.048+0000 7f38b6911700 1 -- 172.21.15.139:0/92830481 shutdown_connections 2021-11-20T06:03:41.037 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.048+0000 7f38b6911700 1 -- 172.21.15.139:0/92830481 wait complete. 2021-11-20T06:03:41.037 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.048+0000 7f38b6911700 1 Processor -- start 2021-11-20T06:03:41.037 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.048+0000 7f38b6911700 1 -- start start 2021-11-20T06:03:41.037 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.048+0000 7f38b6911700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38b00f5930 0x7f38b00f52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:41.037 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.049+0000 7f38b6911700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38b00f6c20 0x7f38b010e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:41.038 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.049+0000 7f38b6911700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b0107a80 con 0x7f38b00f5930 2021-11-20T06:03:41.038 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.049+0000 7f38b6911700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b010e590 con 0x7f38b00f6c20 2021-11-20T06:03:41.038 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.049+0000 7f38affff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38b00f5930 0x7f38b00f52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:41.038 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.049+0000 7f38affff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38b00f5930 0x7f38b00f52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57462/0 (socket says 172.21.15.139:57462) 2021-11-20T06:03:41.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.049+0000 7f38affff700 1 -- 172.21.15.139:0/3709774364 learned_addr learned my addr 172.21.15.139:0/3709774364 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:41.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.049+0000 7f38af7fe700 1 --2- 172.21.15.139:0/3709774364 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38b00f6c20 0x7f38b010e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:41.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.049+0000 7f38affff700 1 -- 172.21.15.139:0/3709774364 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38b00f6c20 msgr2=0x7f38b010e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:41.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.049+0000 7f38affff700 1 --2- 172.21.15.139:0/3709774364 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38b00f6c20 0x7f38b010e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.049+0000 7f38affff700 1 -- 172.21.15.139:0/3709774364 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a4005040 con 0x7f38b00f5930 2021-11-20T06:03:41.040 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.049+0000 7f38affff700 1 --2- 172.21.15.139:0/3709774364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38b00f5930 0x7f38b00f52d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f38a4009f50 tx=0x7f38a4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:41.040 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.049+0000 7f38acff9700 1 -- 172.21.15.139:0/3709774364 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38a4018070 con 0x7f38b00f5930 2021-11-20T06:03:41.040 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.049+0000 7f38b6911700 1 -- 172.21.15.139:0/3709774364 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b010e6d0 con 0x7f38b00f5930 2021-11-20T06:03:41.041 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.049+0000 7f38acff9700 1 -- 172.21.15.139:0/3709774364 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38a4013410 con 0x7f38b00f5930 2021-11-20T06:03:41.042 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.049+0000 7f38acff9700 1 -- 172.21.15.139:0/3709774364 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38a401c3c0 con 0x7f38b00f5930 2021-11-20T06:03:41.043 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.050+0000 7f38b6911700 1 -- 172.21.15.139:0/3709774364 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b010ec00 con 0x7f38b00f5930 2021-11-20T06:03:41.043 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.051+0000 7f38b6911700 1 -- 172.21.15.139:0/3709774364 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b0046330 con 0x7f38b00f5930 2021-11-20T06:03:41.043 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.051+0000 7f38acff9700 1 -- 172.21.15.139:0/3709774364 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f38a401d420 con 0x7f38b00f5930 2021-11-20T06:03:41.043 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.052+0000 7f38acff9700 1 --2- 172.21.15.139:0/3709774364 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f389805c2a0 0x7f389805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:41.044 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.052+0000 7f38acff9700 1 -- 172.21.15.139:0/3709774364 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f38a40776c0 con 0x7f38b00f5930 2021-11-20T06:03:41.044 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.052+0000 7f38af7fe700 1 --2- 172.21.15.139:0/3709774364 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f389805c2a0 0x7f389805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:41.044 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.052+0000 7f38af7fe700 1 --2- 172.21.15.139:0/3709774364 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f389805c2a0 0x7f389805e760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f38a0004c40 tx=0x7f38a000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:41.044 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.055+0000 7f38acff9700 1 -- 172.21.15.139:0/3709774364 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38a404a380 con 0x7f38b00f5930 2021-11-20T06:03:41.192 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:40 smithi178 conmon[36600]: audit 2021-11-20T06:03:40.464260+0000 mon.smithi139 (mon.0) 519 : audit [DBG] from='client.? 172.21.15.139:0/478928574' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2021-11-20T06:03:41.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.214+0000 7f38b6911700 1 -- 172.21.15.139:0/3709774364 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38b0027a00 con 0x7f38b00f5930 2021-11-20T06:03:41.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.215+0000 7f38acff9700 1 -- 172.21.15.139:0/3709774364 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+14123 (secure 0 0 0) 0x7f38a404d9a0 con 0x7f38b00f5930 2021-11-20T06:03:41.200 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:03:41.201 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":41,"fsid":"d819f2cc-49c6-11ec-8c2c-001a4aab830c","created":"2021-11-20T05:58:33.540072+0000","modified":"2021-11-20T06:03:27.205472+0000","last_up_change":"2021-11-20T06:02:27.182023+0000","last_in_change":"2021-11-20T06:02:11.372444+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T06:01:48.405544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2021-11-20T06:03:23.869281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2021-11-20T06:03:24.435074+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fece5a6-7a49-451c-beb8-f90723d42033","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6803","nonce":22698595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6805","nonce":22698595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6809","nonce":22698595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6807","nonce":22698595}]},"public_addr":"172.21.15.139:6803/22698595","cluster_addr":"172.21.15.139:6805/22698595","heartbeat_back_addr":"172.21.15.139:6809/22698595","heartbeat_front_addr":"172.21.15.139:6807/22698595","state":["exists","up"]},{"osd":1,"uuid":"ce18ef72-7c6c-43f7-8a0c-6904f7c9e606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6811","nonce":2530906188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6813","nonce":2530906188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6817","nonce":2530906188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6815","nonce":2530906188}]},"public_addr":"172.21.15.139:6811/2530906188","cluster_addr":"172.21.15.139:6813/2530906188","heartbeat_back_addr":"172.21.15.139:6817/2530906188","heartbeat_front_addr":"172.21.15.139:6815/2530906188","state":["exists","up"]},{"osd":2,"uuid":"80cfe585-e283-45a4-9dd7-f328ec00d21c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6819","nonce":1147042903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6821","nonce":1147042903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6825","nonce":1147042903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6823","nonce":1147042903}]},"public_addr":"172.21.15.139:6819/1147042903","cluster_addr":"172.21.15.139:6821/1147042903","heartbeat_back_addr":"172.21.15.139:6825/1147042903","heartbeat_front_addr":"172.21.15.139:6823/1147042903","state":["exists","up"]},{"osd":3,"uuid":"793e383f-4d5a-41cb-b5b3-b4e4e07d6b8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6801","nonce":927121970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6803","nonce":927121970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6807","nonce":927121970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6805","nonce":927121970}]},"public_addr":"172.21.15.178:6801/927121970","cluster_addr":"172.21.15.178:6803/927121970","heartbeat_back_addr":"172.21.15.178:6807/927121970","heartbeat_front_addr":"172.21.15.178:6805/927121970","state":["exists","up"]},{"osd":4,"uuid":"3e453fbf-8ef9-4227-b3b8-cd6631bff6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6809","nonce":2601954583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6811","nonce":2601954583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6815","nonce":2601954583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6813","nonce":2601954583}]},"public_addr":"172.21.15.178:6809/2601954583","cluster_addr":"172.21.15.178:6811/2601954583","heartbeat_back_addr":"172.21.15.178:6815/2601954583","heartbeat_front_addr":"172.21.15.178:6813/2601954583","state":["exists","up"]},{"osd":5,"uuid":"b7d0eee4-3cb1-43dc-9c3f-f95593cf47cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6817","nonce":3932451665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6819","nonce":3932451665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6823","nonce":3932451665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6821","nonce":3932451665}]},"public_addr":"172.21.15.178:6817/3932451665","cluster_addr":"172.21.15.178:6819/3932451665","heartbeat_back_addr":"172.21.15.178:6823/3932451665","heartbeat_front_addr":"172.21.15.178:6821/3932451665","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:01.290566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:19.499151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:37.169640+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:52.838242+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:02:09.060491+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:02:25.766679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.139:0/3013495710":"2021-11-21T05:59:26.112534+0000","172.21.15.139:0/1461722096":"2021-11-21T05:59:13.525765+0000","172.21.15.139:0/1750602715":"2021-11-21T05:59:26.112534+0000","172.21.15.139:6800/3637001875":"2021-11-21T05:59:26.112534+0000","172.21.15.139:6801/1220659339":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6800/1220659339":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6801/3637001875":"2021-11-21T05:59:26.112534+0000","172.21.15.139:0/2668621147":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6800/4049861874":"2021-11-21T05:58:51.653389+0000","172.21.15.139:0/2863654682":"2021-11-21T05:58:51.653389+0000","172.21.15.139:0/2447016509":"2021-11-21T05:58:51.653389+0000","172.21.15.139:6801/4049861874":"2021-11-21T05:58:51.653389+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T06:03:41.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.217+0000 7f38b6911700 1 -- 172.21.15.139:0/3709774364 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f389805c2a0 msgr2=0x7f389805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:41.203 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.217+0000 7f38b6911700 1 --2- 172.21.15.139:0/3709774364 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f389805c2a0 0x7f389805e760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f38a0004c40 tx=0x7f38a000b040).stop 2021-11-20T06:03:41.203 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.217+0000 7f38b6911700 1 -- 172.21.15.139:0/3709774364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38b00f5930 msgr2=0x7f38b00f52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:41.203 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.217+0000 7f38b6911700 1 --2- 172.21.15.139:0/3709774364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38b00f5930 0x7f38b00f52d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f38a4009f50 tx=0x7f38a4009b30).stop 2021-11-20T06:03:41.203 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.217+0000 7f38b6911700 1 -- 172.21.15.139:0/3709774364 shutdown_connections 2021-11-20T06:03:41.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.217+0000 7f38b6911700 1 --2- 172.21.15.139:0/3709774364 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f389805c2a0 0x7f389805e760 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.217+0000 7f38b6911700 1 --2- 172.21.15.139:0/3709774364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38b00f5930 0x7f38b00f52d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.217+0000 7f38b6911700 1 --2- 172.21.15.139:0/3709774364 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38b00f6c20 0x7f38b010e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.217+0000 7f38b6911700 1 -- 172.21.15.139:0/3709774364 >> 172.21.15.139:0/3709774364 conn(0x7f38b00f0440 msgr2=0x7f38b00fa390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:41.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.217+0000 7f38b6911700 1 -- 172.21.15.139:0/3709774364 shutdown_connections 2021-11-20T06:03:41.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.217+0000 7f38b6911700 1 -- 172.21.15.139:0/3709774364 wait complete. 2021-11-20T06:03:41.214 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:40 smithi139 conmon[33501]: audit 2021-11-20T06:03:40.464260+0000 mon.smithi139 ( 2021-11-20T06:03:41.214 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:40 smithi139 conmon[33501]: mon.0) 519 : audit [DBG] from='client.? 172.21.15.139:0/478928574' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2021-11-20T06:03:41.217 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T06:01:48.405544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2021-11-20T06:03:23.869281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2021-11-20T06:03:24.435074+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-11-20T06:03:41.218 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-11-20T06:03:41.345 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.357+0000 7f033292e700 1 -- 172.21.15.139:0/4231601967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f032c0f69a0 msgr2=0x7f032c0f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:41.345 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.357+0000 7f033292e700 1 --2- 172.21.15.139:0/4231601967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f032c0f69a0 0x7f032c0f6dc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f031c004660 tx=0x7f031c009b30).stop 2021-11-20T06:03:41.345 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.357+0000 7f033292e700 1 -- 172.21.15.139:0/4231601967 shutdown_connections 2021-11-20T06:03:41.346 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.357+0000 7f033292e700 1 --2- 172.21.15.139:0/4231601967 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f032c0f72b0 0x7f032c0f5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.346 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.357+0000 7f033292e700 1 --2- 172.21.15.139:0/4231601967 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f032c0f69a0 0x7f032c0f6dc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.346 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.357+0000 7f033292e700 1 -- 172.21.15.139:0/4231601967 >> 172.21.15.139:0/4231601967 conn(0x7f032c0f0370 msgr2=0x7f032c0f2790 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:41.346 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.357+0000 7f033292e700 1 -- 172.21.15.139:0/4231601967 shutdown_connections 2021-11-20T06:03:41.346 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.357+0000 7f033292e700 1 -- 172.21.15.139:0/4231601967 wait complete. 2021-11-20T06:03:41.347 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.358+0000 7f033292e700 1 Processor -- start 2021-11-20T06:03:41.347 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.358+0000 7f033292e700 1 -- start start 2021-11-20T06:03:41.347 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.358+0000 7f033292e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f032c0f69a0 0x7f032c109ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:41.347 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.358+0000 7f033292e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f032c0f72b0 0x7f032c10bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:41.348 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.358+0000 7f033292e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c0f5430 con 0x7f032c0f69a0 2021-11-20T06:03:41.348 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.358+0000 7f033292e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c0f5570 con 0x7f032c0f72b0 2021-11-20T06:03:41.348 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.358+0000 7f033192c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f032c0f69a0 0x7f032c109ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:41.348 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.358+0000 7f033192c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f032c0f69a0 0x7f032c109ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57472/0 (socket says 172.21.15.139:57472) 2021-11-20T06:03:41.348 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.358+0000 7f033192c700 1 -- 172.21.15.139:0/1403917594 learned_addr learned my addr 172.21.15.139:0/1403917594 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:41.349 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.358+0000 7f033112b700 1 --2- 172.21.15.139:0/1403917594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f032c0f72b0 0x7f032c10bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:41.349 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.359+0000 7f033192c700 1 -- 172.21.15.139:0/1403917594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f032c0f72b0 msgr2=0x7f032c10bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:41.349 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.359+0000 7f033192c700 1 --2- 172.21.15.139:0/1403917594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f032c0f72b0 0x7f032c10bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.349 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.359+0000 7f033192c700 1 -- 172.21.15.139:0/1403917594 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031c005040 con 0x7f032c0f69a0 2021-11-20T06:03:41.350 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.359+0000 7f033192c700 1 --2- 172.21.15.139:0/1403917594 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f032c0f69a0 0x7f032c109ac0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f031c004e30 tx=0x7f031c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:41.350 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.359+0000 7f03227fc700 1 -- 172.21.15.139:0/1403917594 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f031c018070 con 0x7f032c0f69a0 2021-11-20T06:03:41.350 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.359+0000 7f033292e700 1 -- 172.21.15.139:0/1403917594 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032c10c570 con 0x7f032c0f69a0 2021-11-20T06:03:41.350 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.359+0000 7f03227fc700 1 -- 172.21.15.139:0/1403917594 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f031c013410 con 0x7f032c0f69a0 2021-11-20T06:03:41.350 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.359+0000 7f03227fc700 1 -- 172.21.15.139:0/1403917594 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f031c01de20 con 0x7f032c0f69a0 2021-11-20T06:03:41.351 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.359+0000 7f033292e700 1 -- 172.21.15.139:0/1403917594 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f032c10caa0 con 0x7f032c0f69a0 2021-11-20T06:03:41.352 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.360+0000 7f033292e700 1 -- 172.21.15.139:0/1403917594 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f032c0f3fe0 con 0x7f032c0f69a0 2021-11-20T06:03:41.352 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.360+0000 7f03227fc700 1 -- 172.21.15.139:0/1403917594 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f031c01d420 con 0x7f032c0f69a0 2021-11-20T06:03:41.352 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.361+0000 7f03227fc700 1 --2- 172.21.15.139:0/1403917594 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f031805c2b0 0x7f031805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:41.353 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.361+0000 7f03227fc700 1 -- 172.21.15.139:0/1403917594 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f031c077af0 con 0x7f032c0f69a0 2021-11-20T06:03:41.353 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.361+0000 7f033112b700 1 --2- 172.21.15.139:0/1403917594 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f031805c2b0 0x7f031805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:41.353 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.361+0000 7f033112b700 1 --2- 172.21.15.139:0/1403917594 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f031805c2b0 0x7f031805e770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f0328004c40 tx=0x7f032800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:41.353 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.364+0000 7f03227fc700 1 -- 172.21.15.139:0/1403917594 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f031c04a7b0 con 0x7f032c0f69a0 2021-11-20T06:03:41.517 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.532+0000 7f033292e700 1 -- 172.21.15.139:0/1403917594 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f032c045ad0 con 0x7f032c0f69a0 2021-11-20T06:03:41.518 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.532+0000 7f03227fc700 1 -- 172.21.15.139:0/1403917594 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v41) v1 ==== 110+0+10 (secure 0 0 0) 0x7f031c04ddd0 con 0x7f032c0f69a0 2021-11-20T06:03:41.518 INFO:teuthology.orchestra.run.smithi139.stdout:pg_num: 1 2021-11-20T06:03:41.519 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.534+0000 7f033292e700 1 -- 172.21.15.139:0/1403917594 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f031805c2b0 msgr2=0x7f031805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:41.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.534+0000 7f033292e700 1 --2- 172.21.15.139:0/1403917594 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f031805c2b0 0x7f031805e770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f0328004c40 tx=0x7f032800b040).stop 2021-11-20T06:03:41.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.534+0000 7f033292e700 1 -- 172.21.15.139:0/1403917594 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f032c0f69a0 msgr2=0x7f032c109ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:41.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.534+0000 7f033292e700 1 --2- 172.21.15.139:0/1403917594 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f032c0f69a0 0x7f032c109ac0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f031c004e30 tx=0x7f031c009b30).stop 2021-11-20T06:03:41.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.534+0000 7f033292e700 1 -- 172.21.15.139:0/1403917594 shutdown_connections 2021-11-20T06:03:41.521 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.534+0000 7f033292e700 1 --2- 172.21.15.139:0/1403917594 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f031805c2b0 0x7f031805e770 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.521 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.534+0000 7f033292e700 1 --2- 172.21.15.139:0/1403917594 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f032c0f69a0 0x7f032c109ac0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.521 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.534+0000 7f033292e700 1 --2- 172.21.15.139:0/1403917594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f032c0f72b0 0x7f032c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.521 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.534+0000 7f033292e700 1 -- 172.21.15.139:0/1403917594 >> 172.21.15.139:0/1403917594 conn(0x7f032c0f0370 msgr2=0x7f032c0fa1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:41.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.534+0000 7f033292e700 1 -- 172.21.15.139:0/1403917594 shutdown_connections 2021-11-20T06:03:41.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.534+0000 7f033292e700 1 -- 172.21.15.139:0/1403917594 wait complete. 2021-11-20T06:03:41.532 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2021-11-20T06:03:41.656 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.669+0000 7fd3fd86c700 1 -- 172.21.15.139:0/1932431810 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3f80f5580 msgr2=0x7fd3f80f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:41.657 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.669+0000 7fd3fd86c700 1 --2- 172.21.15.139:0/1932431810 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3f80f5580 0x7fd3f80f59a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fd3e8004660 tx=0x7fd3e8009b30).stop 2021-11-20T06:03:41.657 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.669+0000 7fd3fd86c700 1 -- 172.21.15.139:0/1932431810 shutdown_connections 2021-11-20T06:03:41.657 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.669+0000 7fd3fd86c700 1 --2- 172.21.15.139:0/1932431810 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3f80f6820 0x7fd3f80f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.658 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.669+0000 7fd3fd86c700 1 --2- 172.21.15.139:0/1932431810 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3f80f5580 0x7fd3f80f59a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.658 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.669+0000 7fd3fd86c700 1 -- 172.21.15.139:0/1932431810 >> 172.21.15.139:0/1932431810 conn(0x7fd3f80f0ce0 msgr2=0x7fd3f80f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:41.658 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.669+0000 7fd3fd86c700 1 -- 172.21.15.139:0/1932431810 shutdown_connections 2021-11-20T06:03:41.658 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.669+0000 7fd3fd86c700 1 -- 172.21.15.139:0/1932431810 wait complete. 2021-11-20T06:03:41.659 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.670+0000 7fd3fd86c700 1 Processor -- start 2021-11-20T06:03:41.660 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.670+0000 7fd3fd86c700 1 -- start start 2021-11-20T06:03:41.660 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.670+0000 7fd3fd86c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3f80f5580 0x7fd3f8105d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:41.660 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.670+0000 7fd3fd86c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3f80f6820 0x7fd3f8106210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:41.661 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.670+0000 7fd3fd86c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f8106810 con 0x7fd3f80f6820 2021-11-20T06:03:41.661 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.670+0000 7fd3fd86c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f8106950 con 0x7fd3f80f5580 2021-11-20T06:03:41.661 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.670+0000 7fd3f67fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3f80f6820 0x7fd3f8106210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:41.661 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.670+0000 7fd3f67fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3f80f6820 0x7fd3f8106210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57484/0 (socket says 172.21.15.139:57484) 2021-11-20T06:03:41.662 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.670+0000 7fd3f67fc700 1 -- 172.21.15.139:0/270033140 learned_addr learned my addr 172.21.15.139:0/270033140 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:41.662 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.670+0000 7fd3f6ffd700 1 --2- 172.21.15.139:0/270033140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3f80f5580 0x7fd3f8105d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:41.662 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.670+0000 7fd3f67fc700 1 -- 172.21.15.139:0/270033140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3f80f5580 msgr2=0x7fd3f8105d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:41.663 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.670+0000 7fd3f67fc700 1 --2- 172.21.15.139:0/270033140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3f80f5580 0x7fd3f8105d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.663 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.670+0000 7fd3f67fc700 1 -- 172.21.15.139:0/270033140 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e8005040 con 0x7fd3f80f6820 2021-11-20T06:03:41.663 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.670+0000 7fd3f67fc700 1 --2- 172.21.15.139:0/270033140 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3f80f6820 0x7fd3f8106210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fd3ec00e7f0 tx=0x7fd3ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:41.663 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.671+0000 7fd3df7fe700 1 -- 172.21.15.139:0/270033140 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3ec0092e0 con 0x7fd3f80f6820 2021-11-20T06:03:41.664 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.671+0000 7fd3df7fe700 1 -- 172.21.15.139:0/270033140 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd3ec00f040 con 0x7fd3f80f6820 2021-11-20T06:03:41.664 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.671+0000 7fd3fd86c700 1 -- 172.21.15.139:0/270033140 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f810b360 con 0x7fd3f80f6820 2021-11-20T06:03:41.664 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.671+0000 7fd3df7fe700 1 -- 172.21.15.139:0/270033140 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3ec014750 con 0x7fd3f80f6820 2021-11-20T06:03:41.664 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.671+0000 7fd3fd86c700 1 -- 172.21.15.139:0/270033140 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f810b920 con 0x7fd3f80f6820 2021-11-20T06:03:41.665 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.671+0000 7fd3fd86c700 1 -- 172.21.15.139:0/270033140 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f8027a00 con 0x7fd3f80f6820 2021-11-20T06:03:41.666 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.672+0000 7fd3df7fe700 1 -- 172.21.15.139:0/270033140 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd3ec01b030 con 0x7fd3f80f6820 2021-11-20T06:03:41.666 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.672+0000 7fd3df7fe700 1 --2- 172.21.15.139:0/270033140 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd3e005c2a0 0x7fd3e005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:41.666 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.672+0000 7fd3df7fe700 1 -- 172.21.15.139:0/270033140 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fd3ec075c60 con 0x7fd3f80f6820 2021-11-20T06:03:41.666 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.672+0000 7fd3f6ffd700 1 --2- 172.21.15.139:0/270033140 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd3e005c2a0 0x7fd3e005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:41.667 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.673+0000 7fd3f6ffd700 1 --2- 172.21.15.139:0/270033140 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd3e005c2a0 0x7fd3e005e760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fd3e8004630 tx=0x7fd3e8016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:41.667 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.676+0000 7fd3df7fe700 1 -- 172.21.15.139:0/270033140 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3ec0489d0 con 0x7fd3f80f6820 2021-11-20T06:03:41.818 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.832+0000 7fd3fd86c700 1 -- 172.21.15.139:0/270033140 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fd3f80fdd10 con 0x7fd3f80f6820 2021-11-20T06:03:41.819 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.833+0000 7fd3df7fe700 1 -- 172.21.15.139:0/270033140 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7fd3ec019020 con 0x7fd3f80f6820 2021-11-20T06:03:41.819 INFO:teuthology.orchestra.run.smithi139.stdout:pg_num: 32 2021-11-20T06:03:41.820 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.834+0000 7fd3fd86c700 1 -- 172.21.15.139:0/270033140 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd3e005c2a0 msgr2=0x7fd3e005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:41.821 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.834+0000 7fd3fd86c700 1 --2- 172.21.15.139:0/270033140 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd3e005c2a0 0x7fd3e005e760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fd3e8004630 tx=0x7fd3e8016040).stop 2021-11-20T06:03:41.821 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.834+0000 7fd3fd86c700 1 -- 172.21.15.139:0/270033140 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3f80f6820 msgr2=0x7fd3f8106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:41.821 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.834+0000 7fd3fd86c700 1 --2- 172.21.15.139:0/270033140 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3f80f6820 0x7fd3f8106210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fd3ec00e7f0 tx=0x7fd3ec00b040).stop 2021-11-20T06:03:41.821 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.835+0000 7fd3fd86c700 1 -- 172.21.15.139:0/270033140 shutdown_connections 2021-11-20T06:03:41.821 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.835+0000 7fd3fd86c700 1 --2- 172.21.15.139:0/270033140 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd3e005c2a0 0x7fd3e005e760 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.822 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.835+0000 7fd3fd86c700 1 --2- 172.21.15.139:0/270033140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3f80f5580 0x7fd3f8105d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.822 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.835+0000 7fd3fd86c700 1 --2- 172.21.15.139:0/270033140 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd3f80f6820 0x7fd3f8106210 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.822 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.835+0000 7fd3fd86c700 1 -- 172.21.15.139:0/270033140 >> 172.21.15.139:0/270033140 conn(0x7fd3f80f0ce0 msgr2=0x7fd3f80fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:41.823 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.835+0000 7fd3fd86c700 1 -- 172.21.15.139:0/270033140 shutdown_connections 2021-11-20T06:03:41.823 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.835+0000 7fd3fd86c700 1 -- 172.21.15.139:0/270033140 wait complete. 2021-11-20T06:03:41.832 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2021-11-20T06:03:41.962 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.971+0000 7f2d293e2700 1 -- 172.21.15.139:0/1050534125 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d240f5580 msgr2=0x7f2d240f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:41.962 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.971+0000 7f2d293e2700 1 --2- 172.21.15.139:0/1050534125 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d240f5580 0x7f2d240f59a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f2d140099e0 tx=0x7f2d14004d10).stop 2021-11-20T06:03:41.963 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.971+0000 7f2d293e2700 1 -- 172.21.15.139:0/1050534125 shutdown_connections 2021-11-20T06:03:41.963 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.971+0000 7f2d293e2700 1 --2- 172.21.15.139:0/1050534125 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d240f6820 0x7f2d240f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.963 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.971+0000 7f2d293e2700 1 --2- 172.21.15.139:0/1050534125 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d240f5580 0x7f2d240f59a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.963 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.971+0000 7f2d293e2700 1 -- 172.21.15.139:0/1050534125 >> 172.21.15.139:0/1050534125 conn(0x7f2d240f0ce0 msgr2=0x7f2d240f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:41.964 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.971+0000 7f2d293e2700 1 -- 172.21.15.139:0/1050534125 shutdown_connections 2021-11-20T06:03:41.964 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.971+0000 7f2d293e2700 1 -- 172.21.15.139:0/1050534125 wait complete. 2021-11-20T06:03:41.964 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d293e2700 1 Processor -- start 2021-11-20T06:03:41.965 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d293e2700 1 -- start start 2021-11-20T06:03:41.965 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d293e2700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d240f5580 0x7f2d24106f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:41.965 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d293e2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d240f6820 0x7f2d24107470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:41.966 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d293e2700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d24107a20 con 0x7f2d240f6820 2021-11-20T06:03:41.967 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d293e2700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d24107b60 con 0x7f2d240f5580 2021-11-20T06:03:41.967 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d227fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d240f6820 0x7f2d24107470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:41.967 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d227fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d240f6820 0x7f2d24107470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57494/0 (socket says 172.21.15.139:57494) 2021-11-20T06:03:41.968 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d227fc700 1 -- 172.21.15.139:0/830769152 learned_addr learned my addr 172.21.15.139:0/830769152 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:41.968 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d22ffd700 1 --2- 172.21.15.139:0/830769152 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d240f5580 0x7f2d24106f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:41.969 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d227fc700 1 -- 172.21.15.139:0/830769152 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d240f5580 msgr2=0x7f2d24106f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:41.969 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d227fc700 1 --2- 172.21.15.139:0/830769152 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d240f5580 0x7f2d24106f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:41.969 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d227fc700 1 -- 172.21.15.139:0/830769152 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d14009640 con 0x7f2d240f6820 2021-11-20T06:03:41.970 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d227fc700 1 --2- 172.21.15.139:0/830769152 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d240f6820 0x7f2d24107470 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f2d1800c530 tx=0x7f2d18007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:41.970 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d0b7fe700 1 -- 172.21.15.139:0/830769152 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d1800f650 con 0x7f2d240f6820 2021-11-20T06:03:41.971 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d293e2700 1 -- 172.21.15.139:0/830769152 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2410c600 con 0x7f2d240f6820 2021-11-20T06:03:41.971 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d0b7fe700 1 -- 172.21.15.139:0/830769152 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d1800fc60 con 0x7f2d240f6820 2021-11-20T06:03:41.972 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d0b7fe700 1 -- 172.21.15.139:0/830769152 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d18012d30 con 0x7f2d240f6820 2021-11-20T06:03:41.972 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.972+0000 7f2d293e2700 1 -- 172.21.15.139:0/830769152 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2410cb40 con 0x7f2d240f6820 2021-11-20T06:03:41.972 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.973+0000 7f2d293e2700 1 -- 172.21.15.139:0/830769152 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d240fdd10 con 0x7f2d240f6820 2021-11-20T06:03:41.973 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.974+0000 7f2d0b7fe700 1 -- 172.21.15.139:0/830769152 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2d18010070 con 0x7f2d240f6820 2021-11-20T06:03:41.973 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.974+0000 7f2d0b7fe700 1 --2- 172.21.15.139:0/830769152 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f2d0c05c2a0 0x7f2d0c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:41.973 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.974+0000 7f2d0b7fe700 1 -- 172.21.15.139:0/830769152 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f2d18075c70 con 0x7f2d240f6820 2021-11-20T06:03:41.974 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.974+0000 7f2d22ffd700 1 --2- 172.21.15.139:0/830769152 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f2d0c05c2a0 0x7f2d0c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:41.974 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.974+0000 7f2d22ffd700 1 --2- 172.21.15.139:0/830769152 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f2d0c05c2a0 0x7f2d0c05e760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f2d14000f80 tx=0x7f2d14004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:41.975 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:41.978+0000 7f2d0b7fe700 1 -- 172.21.15.139:0/830769152 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d180452b0 con 0x7f2d240f6820 2021-11-20T06:03:42.125 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.139+0000 7f2d293e2700 1 -- 172.21.15.139:0/830769152 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f2d2410d240 con 0x7f2d240f6820 2021-11-20T06:03:42.125 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.139+0000 7f2d0b7fe700 1 -- 172.21.15.139:0/830769152 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2d18048930 con 0x7f2d240f6820 2021-11-20T06:03:42.126 INFO:teuthology.orchestra.run.smithi139.stdout:pg_num: 32 2021-11-20T06:03:42.129 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.141+0000 7f2d293e2700 1 -- 172.21.15.139:0/830769152 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f2d0c05c2a0 msgr2=0x7f2d0c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:42.129 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.141+0000 7f2d293e2700 1 --2- 172.21.15.139:0/830769152 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f2d0c05c2a0 0x7f2d0c05e760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f2d14000f80 tx=0x7f2d14004d10).stop 2021-11-20T06:03:42.129 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.141+0000 7f2d293e2700 1 -- 172.21.15.139:0/830769152 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d240f6820 msgr2=0x7f2d24107470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:42.130 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.141+0000 7f2d293e2700 1 --2- 172.21.15.139:0/830769152 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d240f6820 0x7f2d24107470 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f2d1800c530 tx=0x7f2d18007e20).stop 2021-11-20T06:03:42.130 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.141+0000 7f2d293e2700 1 -- 172.21.15.139:0/830769152 shutdown_connections 2021-11-20T06:03:42.131 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.141+0000 7f2d293e2700 1 --2- 172.21.15.139:0/830769152 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f2d0c05c2a0 0x7f2d0c05e760 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:42.132 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.141+0000 7f2d293e2700 1 --2- 172.21.15.139:0/830769152 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d240f5580 0x7f2d24106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:42.133 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.141+0000 7f2d293e2700 1 --2- 172.21.15.139:0/830769152 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2d240f6820 0x7f2d24107470 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:42.133 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.141+0000 7f2d293e2700 1 -- 172.21.15.139:0/830769152 >> 172.21.15.139:0/830769152 conn(0x7f2d240f0ce0 msgr2=0x7f2d2405f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:42.133 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.141+0000 7f2d293e2700 1 -- 172.21.15.139:0/830769152 shutdown_connections 2021-11-20T06:03:42.134 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.141+0000 7f2d293e2700 1 -- 172.21.15.139:0/830769152 wait complete. 2021-11-20T06:03:42.137 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:41 smithi139 conmon[33501]: cluster 2021-11-20T06:03:40.211184+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:03:42.138 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:41 smithi139 conmon[33501]: 14162) 196 : cluster [DBG] pgmap v157: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 307 B/s wr, 3 op/s 2021-11-20T06:03:42.138 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:41 smithi139 conmon[33501]: audit 2021-11-20T06:03:41.215408+0000 mon.smithi139 (mon.0) 520 : audit [DBG] from='client.? 172.21.15.139:0/3709774364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T06:03:42.139 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:41 smithi139 conmon[33501]: audit 2021-11-20T06:03:41.315744+0000 mon.smithi139 (mon.0) 521 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:42.139 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:41 smithi139 conmon[33501]: audit 2021-11-20T06:03:41.327217+0000 mon.smithi139 (mon.0) 522 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:42.140 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:41 smithi139 conmon[33501]: audit 2021-11-20T06:03:41.330345+0000 mon.smithi139 (mon.0) 523 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:03:42.140 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:41 smithi139 conmon[33501]: audit 2021-11-20T06:03:41.533151+0000 mon.smithi139 (mon.0) 524 : audit [DBG] from='client.? 172.21.15.139:0/1403917594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T06:03:42.140 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:41 smithi139 conmon[33501]: audit 2021-11-20T06:03:41.833765+0000 mon.smithi139 (mon.0) 525 : audit [DBG] from='client.? 172.21.15.139:0/270033140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T06:03:42.143 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:03:42.192 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: cluster 2021-11-20T06:03:40.211184+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:03:42.192 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: 196 : cluster [DBG] pgmap v157: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 307 B/s wr, 3 op/s 2021-11-20T06:03:42.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: audit 2021-11-20T06:03:41.215408+0000 mon.smithi139 (mon.0) 520 : audit [DBG] from='client.? 172.21.15.139:0/3709774364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T06:03:42.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: audit 2021-11-20T06:03:41.315744+0000 mon.smithi139 (mon.0 2021-11-20T06:03:42.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: ) 521 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:42.194 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: audit 2021-11-20T06 2021-11-20T06:03:42.195 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: :03:41.327217+0000 mon.smithi139 (mon.0) 2021-11-20T06:03:42.195 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: 522 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:42.195 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: audit 2021-11-20T 2021-11-20T06:03:42.196 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: 06:03:41.330345+0000 mon.smithi139 (mon. 2021-11-20T06:03:42.196 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: 0) 523 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:03:42.197 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: audit 2021-11-20 2021-11-20T06:03:42.197 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: T06:03:41.533151+0000 mon.smithi139 (mon 2021-11-20T06:03:42.198 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: .0) 524 : audit [DBG] from='client.? 172.21.15.139:0/1403917594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T06:03:42.198 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: audit 2021-11 2021-11-20T06:03:42.199 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: -20T06:03:41.833765+0000 mon.smithi139 2021-11-20T06:03:42.199 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:41 smithi178 conmon[36600]: (mon.0) 525 : audit [DBG] from='client.? 172.21.15.139:0/270033140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T06:03:42.278 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.291+0000 7f781ff99700 1 -- 172.21.15.139:0/4197100549 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78180f6820 msgr2=0x7f78180f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:42.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.291+0000 7f781ff99700 1 --2- 172.21.15.139:0/4197100549 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78180f6820 0x7f78180f6c80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f7808004660 tx=0x7f7808009b30).stop 2021-11-20T06:03:42.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.291+0000 7f781ff99700 1 -- 172.21.15.139:0/4197100549 shutdown_connections 2021-11-20T06:03:42.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.291+0000 7f781ff99700 1 --2- 172.21.15.139:0/4197100549 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78180f6820 0x7f78180f6c80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:42.280 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.291+0000 7f781ff99700 1 --2- 172.21.15.139:0/4197100549 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78180f5580 0x7f78180f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:42.280 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.291+0000 7f781ff99700 1 -- 172.21.15.139:0/4197100549 >> 172.21.15.139:0/4197100549 conn(0x7f78180f0ce0 msgr2=0x7f78180f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:42.280 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.291+0000 7f781ff99700 1 -- 172.21.15.139:0/4197100549 shutdown_connections 2021-11-20T06:03:42.280 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.291+0000 7f781ff99700 1 -- 172.21.15.139:0/4197100549 wait complete. 2021-11-20T06:03:42.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.292+0000 7f781ff99700 1 Processor -- start 2021-11-20T06:03:42.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.292+0000 7f781ff99700 1 -- start start 2021-11-20T06:03:42.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.292+0000 7f781ff99700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78180f5580 0x7f7818105d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:42.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.292+0000 7f781ff99700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78180f6820 0x7f7818106230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:42.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.292+0000 7f781ff99700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78181067e0 con 0x7f78180f5580 2021-11-20T06:03:42.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.292+0000 7f781ff99700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7818106920 con 0x7f78180f6820 2021-11-20T06:03:42.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.292+0000 7f781d534700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78180f6820 0x7f7818106230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:42.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.292+0000 7f781dd35700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78180f5580 0x7f7818105d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:42.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.292+0000 7f781d534700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78180f6820 0x7f7818106230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.139:56218/0 (socket says 172.21.15.139:56218) 2021-11-20T06:03:42.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.292+0000 7f781dd35700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78180f5580 0x7f7818105d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57502/0 (socket says 172.21.15.139:57502) 2021-11-20T06:03:42.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.292+0000 7f781d534700 1 -- 172.21.15.139:0/1218167122 learned_addr learned my addr 172.21.15.139:0/1218167122 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:42.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.292+0000 7f781dd35700 1 -- 172.21.15.139:0/1218167122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78180f6820 msgr2=0x7f7818106230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:42.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.292+0000 7f781dd35700 1 --2- 172.21.15.139:0/1218167122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78180f6820 0x7f7818106230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:42.284 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.292+0000 7f781dd35700 1 -- 172.21.15.139:0/1218167122 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7808005040 con 0x7f78180f5580 2021-11-20T06:03:42.284 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.292+0000 7f781dd35700 1 --2- 172.21.15.139:0/1218167122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78180f5580 0x7f7818105d40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f78180f6680 tx=0x7f781400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:42.284 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.293+0000 7f780e7fc700 1 -- 172.21.15.139:0/1218167122 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78140092e0 con 0x7f78180f5580 2021-11-20T06:03:42.284 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.293+0000 7f781ff99700 1 -- 172.21.15.139:0/1218167122 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781810b3c0 con 0x7f78180f5580 2021-11-20T06:03:42.285 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.293+0000 7f780e7fc700 1 -- 172.21.15.139:0/1218167122 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f781400f040 con 0x7f78180f5580 2021-11-20T06:03:42.285 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.293+0000 7f780e7fc700 1 -- 172.21.15.139:0/1218167122 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78140147a0 con 0x7f78180f5580 2021-11-20T06:03:42.285 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.293+0000 7f781ff99700 1 -- 172.21.15.139:0/1218167122 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f781810b980 con 0x7f78180f5580 2021-11-20T06:03:42.286 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.294+0000 7f780e7fc700 1 -- 172.21.15.139:0/1218167122 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f781401b030 con 0x7f78180f5580 2021-11-20T06:03:42.286 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.294+0000 7f781ff99700 1 -- 172.21.15.139:0/1218167122 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7818046330 con 0x7f78180f5580 2021-11-20T06:03:42.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.294+0000 7f780e7fc700 1 --2- 172.21.15.139:0/1218167122 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f780405c2a0 0x7f780405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:42.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.294+0000 7f780e7fc700 1 -- 172.21.15.139:0/1218167122 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f7814075cb0 con 0x7f78180f5580 2021-11-20T06:03:42.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.294+0000 7f781d534700 1 --2- 172.21.15.139:0/1218167122 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f780405c2a0 0x7f780405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:42.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.295+0000 7f781d534700 1 --2- 172.21.15.139:0/1218167122 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f780405c2a0 0x7f780405e760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f7808009d60 tx=0x7f7808016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:42.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.301+0000 7f780e7fc700 1 -- 172.21.15.139:0/1218167122 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7814048970 con 0x7f78180f5580 2021-11-20T06:03:42.484 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.499+0000 7f781ff99700 1 -- 172.21.15.139:0/1218167122 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7818027a00 con 0x7f78180f5580 2021-11-20T06:03:42.485 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.500+0000 7f780e7fc700 1 -- 172.21.15.139:0/1218167122 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2946 (secure 0 0 0) 0x7f7814019090 con 0x7f78180f5580 2021-11-20T06:03:42.486 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:03:42.486 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14442,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6829/2967963544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2967963544},{"type":"v1","addr":"172.21.15.139:6829","nonce":2967963544}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13}],"filesystems":[{"mdsmap":{"epoch":11,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:03:34.486673+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24237},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14426":{"gid":14426,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.139:6827/724620583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":724620583},{"type":"v1","addr":"172.21.15.139:6827","nonce":724620583}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24237":{"gid":24237,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.178:6825/1867304351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1867304351},{"type":"v1","addr":"172.21.15.178:6825","nonce":1867304351}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:03:42.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.501+0000 7f781ff99700 1 -- 172.21.15.139:0/1218167122 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f780405c2a0 msgr2=0x7f780405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:42.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.501+0000 7f781ff99700 1 --2- 172.21.15.139:0/1218167122 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f780405c2a0 0x7f780405e760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f7808009d60 tx=0x7f7808016040).stop 2021-11-20T06:03:42.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.501+0000 7f781ff99700 1 -- 172.21.15.139:0/1218167122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78180f5580 msgr2=0x7f7818105d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:42.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.501+0000 7f781ff99700 1 --2- 172.21.15.139:0/1218167122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78180f5580 0x7f7818105d40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f78180f6680 tx=0x7f781400b040).stop 2021-11-20T06:03:42.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.501+0000 7f781ff99700 1 -- 172.21.15.139:0/1218167122 shutdown_connections 2021-11-20T06:03:42.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.501+0000 7f781ff99700 1 --2- 172.21.15.139:0/1218167122 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f780405c2a0 0x7f780405e760 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:42.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.501+0000 7f781ff99700 1 --2- 172.21.15.139:0/1218167122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f78180f5580 0x7f7818105d40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:42.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.501+0000 7f781ff99700 1 --2- 172.21.15.139:0/1218167122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78180f6820 0x7f7818106230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:42.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.501+0000 7f781ff99700 1 -- 172.21.15.139:0/1218167122 >> 172.21.15.139:0/1218167122 conn(0x7f78180f0ce0 msgr2=0x7f78180fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:42.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.502+0000 7f781ff99700 1 -- 172.21.15.139:0/1218167122 shutdown_connections 2021-11-20T06:03:42.490 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:42.502+0000 7f781ff99700 1 -- 172.21.15.139:0/1218167122 wait complete. 2021-11-20T06:03:42.490 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 13 2021-11-20T06:03:42.499 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 11, 'max_mds': 1, 'flags': 50} 2021-11-20T06:03:42.499 INFO:teuthology.run_tasks:Running task kclient... 2021-11-20T06:03:42.525 INFO:tasks.kclient:Mounting kernel clients... 2021-11-20T06:03:42.525 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2021-11-20T06:03:42.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:42.526 DEBUG:teuthology.orchestra.run.smithi139:> ip netns list 2021-11-20T06:03:42.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:42.543 DEBUG:teuthology.orchestra.run.smithi139:> sudo ip link delete ceph-brx 2021-11-20T06:03:42.610 INFO:teuthology.orchestra.run.smithi139.stderr:Cannot find device "ceph-brx" 2021-11-20T06:03:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T06:03:42.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:42.613 DEBUG:teuthology.orchestra.run.smithi178:> ip netns list 2021-11-20T06:03:42.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:42.632 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link delete ceph-brx 2021-11-20T06:03:42.703 INFO:teuthology.orchestra.run.smithi178.stderr:Cannot find device "ceph-brx" 2021-11-20T06:03:42.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T06:03:42.704 INFO:tasks.kclient:client.0 config is {} 2021-11-20T06:03:42.705 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2021-11-20T06:03:42.705 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi139.front.sepia.ceph.com 2021-11-20T06:03:42.705 INFO:tasks.cephfs.mount:self.client.name = client.0 2021-11-20T06:03:42.706 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2021-11-20T06:03:42.706 INFO:tasks.cephfs.mount:self.cephfs_name = None 2021-11-20T06:03:42.706 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2021-11-20T06:03:42.706 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2021-11-20T06:03:42.707 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2021-11-20T06:03:42.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:42.708 DEBUG:teuthology.orchestra.run.smithi139:> ip addr 2021-11-20T06:03:42.724 INFO:teuthology.orchestra.run.smithi139.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2021-11-20T06:03:42.724 INFO:teuthology.orchestra.run.smithi139.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2021-11-20T06:03:42.724 INFO:teuthology.orchestra.run.smithi139.stdout: inet 127.0.0.1/8 scope host lo 2021-11-20T06:03:42.725 INFO:teuthology.orchestra.run.smithi139.stdout: valid_lft forever preferred_lft forever 2021-11-20T06:03:42.725 INFO:teuthology.orchestra.run.smithi139.stdout: inet6 ::1/128 scope host 2021-11-20T06:03:42.725 INFO:teuthology.orchestra.run.smithi139.stdout: valid_lft forever preferred_lft forever 2021-11-20T06:03:42.725 INFO:teuthology.orchestra.run.smithi139.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T06:03:42.726 INFO:teuthology.orchestra.run.smithi139.stdout: link/ether 0c:c4:7a:d9:36:e8 brd ff:ff:ff:ff:ff:ff 2021-11-20T06:03:42.726 INFO:teuthology.orchestra.run.smithi139.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T06:03:42.726 INFO:teuthology.orchestra.run.smithi139.stdout: link/ether 0c:c4:7a:d9:36:e9 brd ff:ff:ff:ff:ff:ff 2021-11-20T06:03:42.726 INFO:teuthology.orchestra.run.smithi139.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T06:03:42.726 INFO:teuthology.orchestra.run.smithi139.stdout: link/ether 0c:c4:7a:88:7f:58 brd ff:ff:ff:ff:ff:ff 2021-11-20T06:03:42.727 INFO:teuthology.orchestra.run.smithi139.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2021-11-20T06:03:42.727 INFO:teuthology.orchestra.run.smithi139.stdout: link/ether 0c:c4:7a:88:7f:59 brd ff:ff:ff:ff:ff:ff 2021-11-20T06:03:42.727 INFO:teuthology.orchestra.run.smithi139.stdout: inet 172.21.15.139/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2021-11-20T06:03:42.727 INFO:teuthology.orchestra.run.smithi139.stdout: valid_lft 42269sec preferred_lft 42269sec 2021-11-20T06:03:42.728 INFO:teuthology.orchestra.run.smithi139.stdout: inet6 fe80::ec4:7aff:fe88:7f59/64 scope link noprefixroute 2021-11-20T06:03:42.728 INFO:teuthology.orchestra.run.smithi139.stdout: valid_lft forever preferred_lft forever 2021-11-20T06:03:42.728 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2021-11-20T06:03:42.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:42.729 DEBUG:teuthology.orchestra.run.smithi139:> (cd / && exec bash -c ' 2021-11-20T06:03:42.729 DEBUG:teuthology.orchestra.run.smithi139:> set -e 2021-11-20T06:03:42.730 DEBUG:teuthology.orchestra.run.smithi139:> sudo ip link add name ceph-brx type bridge 2021-11-20T06:03:42.730 DEBUG:teuthology.orchestra.run.smithi139:> sudo ip addr flush dev ceph-brx 2021-11-20T06:03:42.730 DEBUG:teuthology.orchestra.run.smithi139:> sudo ip link set ceph-brx up 2021-11-20T06:03:42.731 DEBUG:teuthology.orchestra.run.smithi139:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2021-11-20T06:03:42.731 DEBUG:teuthology.orchestra.run.smithi139:> ') 2021-11-20T06:03:42.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:42.924 DEBUG:teuthology.orchestra.run.smithi139:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2021-11-20T06:03:42.966 INFO:teuthology.orchestra.run.smithi139.stdout:1 2021-11-20T06:03:42.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:42.967 DEBUG:teuthology.orchestra.run.smithi139:> route 2021-11-20T06:03:43.051 INFO:teuthology.orchestra.run.smithi139.stdout:Kernel IP routing table 2021-11-20T06:03:43.052 INFO:teuthology.orchestra.run.smithi139.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2021-11-20T06:03:43.052 INFO:teuthology.orchestra.run.smithi139.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2021-11-20T06:03:43.052 INFO:teuthology.orchestra.run.smithi139.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2021-11-20T06:03:43.052 INFO:teuthology.orchestra.run.smithi139.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2021-11-20T06:03:43.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:43.053 DEBUG:teuthology.orchestra.run.smithi139:> (cd / && exec bash -c ' 2021-11-20T06:03:43.054 DEBUG:teuthology.orchestra.run.smithi139:> set -e 2021-11-20T06:03:43.054 DEBUG:teuthology.orchestra.run.smithi139:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2021-11-20T06:03:43.054 DEBUG:teuthology.orchestra.run.smithi139:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2021-11-20T06:03:43.055 DEBUG:teuthology.orchestra.run.smithi139:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2021-11-20T06:03:43.055 DEBUG:teuthology.orchestra.run.smithi139:> ') 2021-11-20T06:03:43.128 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:42 smithi139 conmon[33501]: audit 2021-11-20T06:03:42.140394+0000 mon.smithi139 (mon.0) 526 : audit [DBG] from='client.? 172.21.15.139:0/830769152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T06:03:43.128 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:42 smithi139 conmon[33501]: audit 2021-11-20T06:03:42.500547+0000 mon.smithi139 (mon.0) 527 : audit [DBG] from='client.? 172.21.15.139:0/1218167122' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:03:43.191 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:42 smithi178 conmon[36600]: audit 2021-11-20T06:03:42.140394+0000 mon.smithi139 (mon.0) 526 : audit [DBG] from='client.? 172.21.15.139:0/830769152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T06:03:43.192 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:42 smithi178 conmon[36600]: audit 2021-11-20T06:03:42.500547+0000 mon.smithi139 (mon.0) 527 : audit [DBG] from='client.? 172.21.15.139:0/1218167122' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:03:43.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:43.370 DEBUG:teuthology.orchestra.run.smithi139:> ip netns list 2021-11-20T06:03:43.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:43.428 DEBUG:teuthology.orchestra.run.smithi139:> ip netns list-id 2021-11-20T06:03:43.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:43.483 DEBUG:teuthology.orchestra.run.smithi139:> (cd / && exec bash -c ' 2021-11-20T06:03:43.483 DEBUG:teuthology.orchestra.run.smithi139:> set -e 2021-11-20T06:03:43.484 DEBUG:teuthology.orchestra.run.smithi139:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2021-11-20T06:03:43.484 DEBUG:teuthology.orchestra.run.smithi139:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2021-11-20T06:03:43.484 DEBUG:teuthology.orchestra.run.smithi139:> ') 2021-11-20T06:03:43.576 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2021-11-20T06:03:43.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:43.577 DEBUG:teuthology.orchestra.run.smithi139:> (cd / && exec bash -c ' 2021-11-20T06:03:43.578 DEBUG:teuthology.orchestra.run.smithi139:> set -e 2021-11-20T06:03:43.578 DEBUG:teuthology.orchestra.run.smithi139:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2021-11-20T06:03:43.578 DEBUG:teuthology.orchestra.run.smithi139:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2021-11-20T06:03:43.579 DEBUG:teuthology.orchestra.run.smithi139:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2021-11-20T06:03:43.579 DEBUG:teuthology.orchestra.run.smithi139:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2021-11-20T06:03:43.579 DEBUG:teuthology.orchestra.run.smithi139:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2021-11-20T06:03:43.580 DEBUG:teuthology.orchestra.run.smithi139:> ') 2021-11-20T06:03:43.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:43.780 DEBUG:teuthology.orchestra.run.smithi139:> (cd / && exec bash -c ' 2021-11-20T06:03:43.780 DEBUG:teuthology.orchestra.run.smithi139:> set -e 2021-11-20T06:03:43.781 DEBUG:teuthology.orchestra.run.smithi139:> sudo ip link set brx.0 up 2021-11-20T06:03:43.781 DEBUG:teuthology.orchestra.run.smithi139:> sudo ip link set dev brx.0 master ceph-brx 2021-11-20T06:03:43.781 DEBUG:teuthology.orchestra.run.smithi139:> ') 2021-11-20T06:03:43.905 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-11-20T06:03:44.035 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.048+0000 7fedd31fa700 1 -- 172.21.15.139:0/3518978932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedcc0f7870 msgr2=0x7fedcc0f7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:44.036 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.048+0000 7fedd31fa700 1 --2- 172.21.15.139:0/3518978932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedcc0f7870 0x7fedcc0f7cd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fedbc004660 tx=0x7fedbc009b30).stop 2021-11-20T06:03:44.036 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.048+0000 7fedd31fa700 1 -- 172.21.15.139:0/3518978932 shutdown_connections 2021-11-20T06:03:44.036 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.048+0000 7fedd31fa700 1 --2- 172.21.15.139:0/3518978932 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedcc0f7870 0x7fedcc0f7cd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.036 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.048+0000 7fedd31fa700 1 --2- 172.21.15.139:0/3518978932 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedcc0f65d0 0x7fedcc0f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.037 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.048+0000 7fedd31fa700 1 -- 172.21.15.139:0/3518978932 >> 172.21.15.139:0/3518978932 conn(0x7fedcc0f1d10 msgr2=0x7fedcc0f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:44.037 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.048+0000 7fedd31fa700 1 -- 172.21.15.139:0/3518978932 shutdown_connections 2021-11-20T06:03:44.037 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.048+0000 7fedd31fa700 1 -- 172.21.15.139:0/3518978932 wait complete. 2021-11-20T06:03:44.037 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.048+0000 7fedd31fa700 1 Processor -- start 2021-11-20T06:03:44.038 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.048+0000 7fedd31fa700 1 -- start start 2021-11-20T06:03:44.038 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedd31fa700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedcc0f65d0 0x7fedcc106d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:44.038 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedd31fa700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedcc0f7870 0x7fedcc107280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:44.038 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedd31fa700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc107880 con 0x7fedcc0f7870 2021-11-20T06:03:44.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedd31fa700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc108090 con 0x7fedcc0f65d0 2021-11-20T06:03:44.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedd19f7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedcc0f7870 0x7fedcc107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:44.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedd19f7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedcc0f7870 0x7fedcc107280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57514/0 (socket says 172.21.15.139:57514) 2021-11-20T06:03:44.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedd19f7700 1 -- 172.21.15.139:0/1827783017 learned_addr learned my addr 172.21.15.139:0/1827783017 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:44.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedd21f8700 1 --2- 172.21.15.139:0/1827783017 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedcc0f65d0 0x7fedcc106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:44.040 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedd19f7700 1 -- 172.21.15.139:0/1827783017 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedcc0f65d0 msgr2=0x7fedcc106d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:44.040 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedd19f7700 1 --2- 172.21.15.139:0/1827783017 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedcc0f65d0 0x7fedcc106d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.040 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedd19f7700 1 -- 172.21.15.139:0/1827783017 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc005040 con 0x7fedcc0f7870 2021-11-20T06:03:44.040 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedd19f7700 1 --2- 172.21.15.139:0/1827783017 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedcc0f7870 0x7fedcc107280 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fedbc009f50 tx=0x7fedbc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:44.041 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedc2ffd700 1 -- 172.21.15.139:0/1827783017 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedbc018070 con 0x7fedcc0f7870 2021-11-20T06:03:44.041 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedc2ffd700 1 -- 172.21.15.139:0/1827783017 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fedbc013410 con 0x7fedcc0f7870 2021-11-20T06:03:44.041 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedd31fa700 1 -- 172.21.15.139:0/1827783017 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedcc1082b0 con 0x7fedcc0f7870 2021-11-20T06:03:44.041 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedc2ffd700 1 -- 172.21.15.139:0/1827783017 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedbc01c3c0 con 0x7fedcc0f7870 2021-11-20T06:03:44.042 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.049+0000 7fedd31fa700 1 -- 172.21.15.139:0/1827783017 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedcc108810 con 0x7fedcc0f7870 2021-11-20T06:03:44.042 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.050+0000 7fedd31fa700 1 -- 172.21.15.139:0/1827783017 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedcc0f59c0 con 0x7fedcc0f7870 2021-11-20T06:03:44.043 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.051+0000 7fedc2ffd700 1 -- 172.21.15.139:0/1827783017 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fedbc01d420 con 0x7fedcc0f7870 2021-11-20T06:03:44.043 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.051+0000 7fedc2ffd700 1 --2- 172.21.15.139:0/1827783017 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fedb805c300 0x7fedb805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:44.044 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.051+0000 7fedc2ffd700 1 -- 172.21.15.139:0/1827783017 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fedbc077a70 con 0x7fedcc0f7870 2021-11-20T06:03:44.044 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.051+0000 7fedd21f8700 1 --2- 172.21.15.139:0/1827783017 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fedb805c300 0x7fedb805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:44.044 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.051+0000 7fedd21f8700 1 --2- 172.21.15.139:0/1827783017 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fedb805c300 0x7fedb805e7c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fedcc106b00 tx=0x7fedc800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:44.044 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.055+0000 7fedc2ffd700 1 -- 172.21.15.139:0/1827783017 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fedbc04a7b0 con 0x7fedcc0f7870 2021-11-20T06:03:44.192 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:43 smithi178 conmon[36600]: cluster 2021-11-20T06:03:42.211579+0000 mgr.smithi139.dplyxn (mgr.14162) 197 : cluster [DBG] pgmap v158: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 255 B/s wr, 3 op/s 2021-11-20T06:03:44.192 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:43 smithi178 conmon[36600]: cluster 2021-11-20T06:03:43.187004+0000 mon.smithi139 (mon.0) 528 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2967963544,v1:172.21.15.139:6829/2967963544] up:standby 2021-11-20T06:03:44.193 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:43 smithi178 conmon[36600]: cluster 2021-11-20T06:03:43.187050+0000 mon.smithi139 (mon.0) 529 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby-replay 2 up:standby 2021-11-20T06:03:44.195 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.210+0000 7fedd31fa700 1 -- 172.21.15.139:0/1827783017 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedcc0fee80 con 0x7fedcc0f7870 2021-11-20T06:03:44.196 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.211+0000 7fedc2ffd700 1 -- 172.21.15.139:0/1827783017 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+14123 (secure 0 0 0) 0x7fedcc0fee80 con 0x7fedcc0f7870 2021-11-20T06:03:44.197 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:03:44.197 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":41,"fsid":"d819f2cc-49c6-11ec-8c2c-001a4aab830c","created":"2021-11-20T05:58:33.540072+0000","modified":"2021-11-20T06:03:27.205472+0000","last_up_change":"2021-11-20T06:02:27.182023+0000","last_in_change":"2021-11-20T06:02:11.372444+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T06:01:48.405544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2021-11-20T06:03:23.869281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2021-11-20T06:03:24.435074+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fece5a6-7a49-451c-beb8-f90723d42033","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6803","nonce":22698595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6805","nonce":22698595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6809","nonce":22698595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6807","nonce":22698595}]},"public_addr":"172.21.15.139:6803/22698595","cluster_addr":"172.21.15.139:6805/22698595","heartbeat_back_addr":"172.21.15.139:6809/22698595","heartbeat_front_addr":"172.21.15.139:6807/22698595","state":["exists","up"]},{"osd":1,"uuid":"ce18ef72-7c6c-43f7-8a0c-6904f7c9e606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6811","nonce":2530906188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6813","nonce":2530906188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6817","nonce":2530906188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6815","nonce":2530906188}]},"public_addr":"172.21.15.139:6811/2530906188","cluster_addr":"172.21.15.139:6813/2530906188","heartbeat_back_addr":"172.21.15.139:6817/2530906188","heartbeat_front_addr":"172.21.15.139:6815/2530906188","state":["exists","up"]},{"osd":2,"uuid":"80cfe585-e283-45a4-9dd7-f328ec00d21c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6819","nonce":1147042903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6821","nonce":1147042903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6825","nonce":1147042903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6823","nonce":1147042903}]},"public_addr":"172.21.15.139:6819/1147042903","cluster_addr":"172.21.15.139:6821/1147042903","heartbeat_back_addr":"172.21.15.139:6825/1147042903","heartbeat_front_addr":"172.21.15.139:6823/1147042903","state":["exists","up"]},{"osd":3,"uuid":"793e383f-4d5a-41cb-b5b3-b4e4e07d6b8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6801","nonce":927121970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6803","nonce":927121970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6807","nonce":927121970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6805","nonce":927121970}]},"public_addr":"172.21.15.178:6801/927121970","cluster_addr":"172.21.15.178:6803/927121970","heartbeat_back_addr":"172.21.15.178:6807/927121970","heartbeat_front_addr":"172.21.15.178:6805/927121970","state":["exists","up"]},{"osd":4,"uuid":"3e453fbf-8ef9-4227-b3b8-cd6631bff6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6809","nonce":2601954583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6811","nonce":2601954583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6815","nonce":2601954583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6813","nonce":2601954583}]},"public_addr":"172.21.15.178:6809/2601954583","cluster_addr":"172.21.15.178:6811/2601954583","heartbeat_back_addr":"172.21.15.178:6815/2601954583","heartbeat_front_addr":"172.21.15.178:6813/2601954583","state":["exists","up"]},{"osd":5,"uuid":"b7d0eee4-3cb1-43dc-9c3f-f95593cf47cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6817","nonce":3932451665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6819","nonce":3932451665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6823","nonce":3932451665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6821","nonce":3932451665}]},"public_addr":"172.21.15.178:6817/3932451665","cluster_addr":"172.21.15.178:6819/3932451665","heartbeat_back_addr":"172.21.15.178:6823/3932451665","heartbeat_front_addr":"172.21.15.178:6821/3932451665","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:01.290566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:19.499151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:37.169640+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:52.838242+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:02:09.060491+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:02:25.766679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.139:0/3013495710":"2021-11-21T05:59:26.112534+0000","172.21.15.139:0/1461722096":"2021-11-21T05:59:13.525765+0000","172.21.15.139:0/1750602715":"2021-11-21T05:59:26.112534+0000","172.21.15.139:6800/3637001875":"2021-11-21T05:59:26.112534+0000","172.21.15.139:6801/1220659339":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6800/1220659339":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6801/3637001875":"2021-11-21T05:59:26.112534+0000","172.21.15.139:0/2668621147":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6800/4049861874":"2021-11-21T05:58:51.653389+0000","172.21.15.139:0/2863654682":"2021-11-21T05:58:51.653389+0000","172.21.15.139:0/2447016509":"2021-11-21T05:58:51.653389+0000","172.21.15.139:6801/4049861874":"2021-11-21T05:58:51.653389+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T06:03:44.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.213+0000 7fedd31fa700 1 -- 172.21.15.139:0/1827783017 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fedb805c300 msgr2=0x7fedb805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:44.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.213+0000 7fedd31fa700 1 --2- 172.21.15.139:0/1827783017 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fedb805c300 0x7fedb805e7c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fedcc106b00 tx=0x7fedc800b040).stop 2021-11-20T06:03:44.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.213+0000 7fedd31fa700 1 -- 172.21.15.139:0/1827783017 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedcc0f7870 msgr2=0x7fedcc107280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:44.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.213+0000 7fedd31fa700 1 --2- 172.21.15.139:0/1827783017 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedcc0f7870 0x7fedcc107280 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fedbc009f50 tx=0x7fedbc009b30).stop 2021-11-20T06:03:44.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.213+0000 7fedd31fa700 1 -- 172.21.15.139:0/1827783017 shutdown_connections 2021-11-20T06:03:44.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.213+0000 7fedd31fa700 1 --2- 172.21.15.139:0/1827783017 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fedb805c300 0x7fedb805e7c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.213+0000 7fedd31fa700 1 --2- 172.21.15.139:0/1827783017 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedcc0f65d0 0x7fedcc106d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.213+0000 7fedd31fa700 1 --2- 172.21.15.139:0/1827783017 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fedcc0f7870 0x7fedcc107280 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.213+0000 7fedd31fa700 1 -- 172.21.15.139:0/1827783017 >> 172.21.15.139:0/1827783017 conn(0x7fedcc0f1d10 msgr2=0x7fedcc0fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:44.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.213+0000 7fedd31fa700 1 -- 172.21.15.139:0/1827783017 shutdown_connections 2021-11-20T06:03:44.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.214+0000 7fedd31fa700 1 -- 172.21.15.139:0/1827783017 wait complete. 2021-11-20T06:03:44.211 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:43 smithi139 conmon[33501]: cluster 2021-11-20T06:03:42.211579+0000 mgr.smithi139.dplyxn (mgr.14162) 197 : cluster [DBG] pgmap v158: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 255 B/s wr, 3 op/s 2021-11-20T06:03:44.211 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:43 smithi139 conmon[33501]: cluster 2021-11-20T06:03:43.187004+0000 mon.smithi139 (mon.0) 528 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2967963544,v1:172.21.15.139:6829/2967963544] up:standby 2021-11-20T06:03:44.212 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:43 smithi139 conmon[33501]: cluster 2021-11-20T06:03:43.187050+0000 mon.smithi139 (mon.0) 529 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi178.blvkbx=up:active} 1 up:standby-replay 2 up:standby 2021-11-20T06:03:44.213 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T06:01:48.405544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2021-11-20T06:03:23.869281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2021-11-20T06:03:24.435074+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-11-20T06:03:44.214 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-11-20T06:03:44.335 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.348+0000 7f442c2f8700 1 -- 172.21.15.139:0/1954221529 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f44240f6aa0 msgr2=0x7f44240f6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:44.336 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.348+0000 7f442c2f8700 1 --2- 172.21.15.139:0/1954221529 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f44240f6aa0 0x7f44240f6ec0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f4420004660 tx=0x7f4420009b30).stop 2021-11-20T06:03:44.336 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.348+0000 7f442c2f8700 1 -- 172.21.15.139:0/1954221529 shutdown_connections 2021-11-20T06:03:44.336 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.348+0000 7f442c2f8700 1 --2- 172.21.15.139:0/1954221529 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44240f7440 0x7f44240f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.337 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.348+0000 7f442c2f8700 1 --2- 172.21.15.139:0/1954221529 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f44240f6aa0 0x7f44240f6ec0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.337 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.348+0000 7f442c2f8700 1 -- 172.21.15.139:0/1954221529 >> 172.21.15.139:0/1954221529 conn(0x7f44240f03d0 msgr2=0x7f44240f2810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:44.337 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.348+0000 7f442c2f8700 1 -- 172.21.15.139:0/1954221529 shutdown_connections 2021-11-20T06:03:44.337 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.348+0000 7f442c2f8700 1 -- 172.21.15.139:0/1954221529 wait complete. 2021-11-20T06:03:44.338 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.349+0000 7f442c2f8700 1 Processor -- start 2021-11-20T06:03:44.338 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.349+0000 7f442c2f8700 1 -- start start 2021-11-20T06:03:44.338 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.349+0000 7f442c2f8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f44240f6aa0 0x7f4424105c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:44.338 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.349+0000 7f442c2f8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44240f7440 0x7f4424106170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:44.339 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.349+0000 7f442c2f8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4424106770 con 0x7f44240f6aa0 2021-11-20T06:03:44.339 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.349+0000 7f442c2f8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44241068e0 con 0x7f44240f7440 2021-11-20T06:03:44.339 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.349+0000 7f442a094700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f44240f6aa0 0x7f4424105c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:44.339 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.349+0000 7f442a094700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f44240f6aa0 0x7f4424105c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57522/0 (socket says 172.21.15.139:57522) 2021-11-20T06:03:44.340 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.349+0000 7f442a094700 1 -- 172.21.15.139:0/1131473791 learned_addr learned my addr 172.21.15.139:0/1131473791 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:44.340 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.350+0000 7f442a094700 1 -- 172.21.15.139:0/1131473791 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44240f7440 msgr2=0x7f4424106170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:44.340 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.350+0000 7f442a094700 1 --2- 172.21.15.139:0/1131473791 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44240f7440 0x7f4424106170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:03:44.340 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.350+0000 7f442a094700 1 -- 172.21.15.139:0/1131473791 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4420005040 con 0x7f44240f6aa0 2021-11-20T06:03:44.341 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.350+0000 7f442a094700 1 --2- 172.21.15.139:0/1131473791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f44240f6aa0 0x7f4424105c80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f4420004e30 tx=0x7f4420009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:44.341 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.350+0000 7f441affd700 1 -- 172.21.15.139:0/1131473791 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4420018070 con 0x7f44240f6aa0 2021-11-20T06:03:44.341 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.350+0000 7f442c2f8700 1 -- 172.21.15.139:0/1131473791 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f442410b2f0 con 0x7f44240f6aa0 2021-11-20T06:03:44.341 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.350+0000 7f442c2f8700 1 -- 172.21.15.139:0/1131473791 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442410b820 con 0x7f44240f6aa0 2021-11-20T06:03:44.342 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.350+0000 7f441affd700 1 -- 172.21.15.139:0/1131473791 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4420013410 con 0x7f44240f6aa0 2021-11-20T06:03:44.342 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.350+0000 7f441affd700 1 -- 172.21.15.139:0/1131473791 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f442001c3c0 con 0x7f44240f6aa0 2021-11-20T06:03:44.343 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.351+0000 7f442c2f8700 1 -- 172.21.15.139:0/1131473791 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4424046330 con 0x7f44240f6aa0 2021-11-20T06:03:44.344 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.351+0000 7f441affd700 1 -- 172.21.15.139:0/1131473791 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f442001d420 con 0x7f44240f6aa0 2021-11-20T06:03:44.344 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.352+0000 7f441affd700 1 --2- 172.21.15.139:0/1131473791 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f441005c2b0 0x7f441005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:44.344 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.352+0000 7f441affd700 1 -- 172.21.15.139:0/1131473791 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f4420077b10 con 0x7f44240f6aa0 2021-11-20T06:03:44.344 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.352+0000 7f4429893700 1 --2- 172.21.15.139:0/1131473791 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f441005c2b0 0x7f441005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:44.345 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.352+0000 7f4429893700 1 --2- 172.21.15.139:0/1131473791 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f441005c2b0 0x7f441005e770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f4414009880 tx=0x7f4414009a60).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:44.345 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.356+0000 7f441affd700 1 -- 172.21.15.139:0/1131473791 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f442004a7d0 con 0x7f44240f6aa0 2021-11-20T06:03:44.499 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.514+0000 7f442c2f8700 1 -- 172.21.15.139:0/1131473791 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f44240fdcd0 con 0x7f44240f6aa0 2021-11-20T06:03:44.500 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.515+0000 7f441affd700 1 -- 172.21.15.139:0/1131473791 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v41) v1 ==== 110+0+10 (secure 0 0 0) 0x7f442004ddf0 con 0x7f44240f6aa0 2021-11-20T06:03:44.500 INFO:teuthology.orchestra.run.smithi139.stdout:pg_num: 1 2021-11-20T06:03:44.502 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.516+0000 7f442c2f8700 1 -- 172.21.15.139:0/1131473791 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f441005c2b0 msgr2=0x7f441005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:44.503 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.516+0000 7f442c2f8700 1 --2- 172.21.15.139:0/1131473791 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f441005c2b0 0x7f441005e770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f4414009880 tx=0x7f4414009a60).stop 2021-11-20T06:03:44.503 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.516+0000 7f442c2f8700 1 -- 172.21.15.139:0/1131473791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f44240f6aa0 msgr2=0x7f4424105c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:44.503 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.516+0000 7f442c2f8700 1 --2- 172.21.15.139:0/1131473791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f44240f6aa0 0x7f4424105c80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f4420004e30 tx=0x7f4420009b30).stop 2021-11-20T06:03:44.503 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.516+0000 7f442c2f8700 1 -- 172.21.15.139:0/1131473791 shutdown_connections 2021-11-20T06:03:44.503 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.516+0000 7f442c2f8700 1 --2- 172.21.15.139:0/1131473791 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f441005c2b0 0x7f441005e770 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.504 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.516+0000 7f442c2f8700 1 --2- 172.21.15.139:0/1131473791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f44240f6aa0 0x7f4424105c80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.504 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.516+0000 7f442c2f8700 1 --2- 172.21.15.139:0/1131473791 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44240f7440 0x7f4424106170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:03:44.504 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.516+0000 7f442c2f8700 1 -- 172.21.15.139:0/1131473791 >> 172.21.15.139:0/1131473791 conn(0x7f44240f03d0 msgr2=0x7f44240fa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:44.504 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.517+0000 7f442c2f8700 1 -- 172.21.15.139:0/1131473791 shutdown_connections 2021-11-20T06:03:44.505 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.517+0000 7f442c2f8700 1 -- 172.21.15.139:0/1131473791 wait complete. 2021-11-20T06:03:44.514 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2021-11-20T06:03:44.639 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.650+0000 7f278172b700 1 -- 172.21.15.139:0/1653821618 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f277c0f5b90 msgr2=0x7f277c0f5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:44.639 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.650+0000 7f278172b700 1 --2- 172.21.15.139:0/1653821618 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f277c0f5b90 0x7f277c0f5fb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f276c004660 tx=0x7f276c009b30).stop 2021-11-20T06:03:44.640 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.650+0000 7f278172b700 1 -- 172.21.15.139:0/1653821618 shutdown_connections 2021-11-20T06:03:44.640 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.650+0000 7f278172b700 1 --2- 172.21.15.139:0/1653821618 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f277c0f6530 0x7f277c0f6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.640 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.650+0000 7f278172b700 1 --2- 172.21.15.139:0/1653821618 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f277c0f5b90 0x7f277c0f5fb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.640 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.650+0000 7f278172b700 1 -- 172.21.15.139:0/1653821618 >> 172.21.15.139:0/1653821618 conn(0x7f277c0f04a0 msgr2=0x7f277c0f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:44.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.650+0000 7f278172b700 1 -- 172.21.15.139:0/1653821618 shutdown_connections 2021-11-20T06:03:44.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.650+0000 7f278172b700 1 -- 172.21.15.139:0/1653821618 wait complete. 2021-11-20T06:03:44.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.650+0000 7f278172b700 1 Processor -- start 2021-11-20T06:03:44.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.650+0000 7f278172b700 1 -- start start 2021-11-20T06:03:44.642 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f278172b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f277c0f5b90 0x7f277c0f5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:44.642 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f278172b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f277c0f6530 0x7f277c10bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:44.642 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f278172b700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277c10c1b0 con 0x7f277c0f5b90 2021-11-20T06:03:44.643 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f278172b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277c10c2f0 con 0x7f277c0f6530 2021-11-20T06:03:44.644 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f277affd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f277c0f5b90 0x7f277c0f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:44.644 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f277affd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f277c0f5b90 0x7f277c0f5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57530/0 (socket says 172.21.15.139:57530) 2021-11-20T06:03:44.644 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f277affd700 1 -- 172.21.15.139:0/2026208059 learned_addr learned my addr 172.21.15.139:0/2026208059 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:44.645 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f277a7fc700 1 --2- 172.21.15.139:0/2026208059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f277c0f6530 0x7f277c10bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:44.645 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f277affd700 1 -- 172.21.15.139:0/2026208059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f277c0f6530 msgr2=0x7f277c10bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:44.646 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f277affd700 1 --2- 172.21.15.139:0/2026208059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f277c0f6530 0x7f277c10bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.646 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f277affd700 1 -- 172.21.15.139:0/2026208059 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f276c005040 con 0x7f277c0f5b90 2021-11-20T06:03:44.647 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f277affd700 1 --2- 172.21.15.139:0/2026208059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f277c0f5b90 0x7f277c0f5370 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f276c004e30 tx=0x7f276c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:44.647 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f27637fe700 1 -- 172.21.15.139:0/2026208059 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f276c018070 con 0x7f277c0f5b90 2021-11-20T06:03:44.648 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f27637fe700 1 -- 172.21.15.139:0/2026208059 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f276c013410 con 0x7f277c0f5b90 2021-11-20T06:03:44.648 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f278172b700 1 -- 172.21.15.139:0/2026208059 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f277c10c4f0 con 0x7f277c0f5b90 2021-11-20T06:03:44.649 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.651+0000 7f27637fe700 1 -- 172.21.15.139:0/2026208059 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f276c01de20 con 0x7f277c0f5b90 2021-11-20T06:03:44.649 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.652+0000 7f278172b700 1 -- 172.21.15.139:0/2026208059 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f277c10ca20 con 0x7f277c0f5b90 2021-11-20T06:03:44.650 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.652+0000 7f278172b700 1 -- 172.21.15.139:0/2026208059 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f277c027a00 con 0x7f277c0f5b90 2021-11-20T06:03:44.651 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.653+0000 7f27637fe700 1 -- 172.21.15.139:0/2026208059 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f276c01d420 con 0x7f277c0f5b90 2021-11-20T06:03:44.651 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.653+0000 7f27637fe700 1 --2- 172.21.15.139:0/2026208059 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f276405c2f0 0x7f276405e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:44.651 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.653+0000 7f27637fe700 1 -- 172.21.15.139:0/2026208059 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f276c077c00 con 0x7f277c0f5b90 2021-11-20T06:03:44.652 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.654+0000 7f277a7fc700 1 --2- 172.21.15.139:0/2026208059 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f276405c2f0 0x7f276405e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:44.652 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.654+0000 7f277a7fc700 1 --2- 172.21.15.139:0/2026208059 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f276405c2f0 0x7f276405e7b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2770004c60 tx=0x7f277000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:44.653 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.659+0000 7f27637fe700 1 -- 172.21.15.139:0/2026208059 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f276c04a800 con 0x7f277c0f5b90 2021-11-20T06:03:44.803 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.817+0000 7f278172b700 1 -- 172.21.15.139:0/2026208059 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f277c0fdd70 con 0x7f277c0f5b90 2021-11-20T06:03:44.804 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.818+0000 7f27637fe700 1 -- 172.21.15.139:0/2026208059 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7f276c04de20 con 0x7f277c0f5b90 2021-11-20T06:03:44.804 INFO:teuthology.orchestra.run.smithi139.stdout:pg_num: 32 2021-11-20T06:03:44.806 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.820+0000 7f278172b700 1 -- 172.21.15.139:0/2026208059 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f276405c2f0 msgr2=0x7f276405e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:44.806 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.820+0000 7f278172b700 1 --2- 172.21.15.139:0/2026208059 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f276405c2f0 0x7f276405e7b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2770004c60 tx=0x7f277000b040).stop 2021-11-20T06:03:44.806 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.820+0000 7f278172b700 1 -- 172.21.15.139:0/2026208059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f277c0f5b90 msgr2=0x7f277c0f5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:44.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.820+0000 7f278172b700 1 --2- 172.21.15.139:0/2026208059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f277c0f5b90 0x7f277c0f5370 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f276c004e30 tx=0x7f276c009b30).stop 2021-11-20T06:03:44.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.820+0000 7f278172b700 1 -- 172.21.15.139:0/2026208059 shutdown_connections 2021-11-20T06:03:44.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.820+0000 7f278172b700 1 --2- 172.21.15.139:0/2026208059 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f276405c2f0 0x7f276405e7b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.820+0000 7f278172b700 1 --2- 172.21.15.139:0/2026208059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f277c0f5b90 0x7f277c0f5370 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.808 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.820+0000 7f278172b700 1 --2- 172.21.15.139:0/2026208059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f277c0f6530 0x7f277c10bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.808 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.820+0000 7f278172b700 1 -- 172.21.15.139:0/2026208059 >> 172.21.15.139:0/2026208059 conn(0x7f277c0f04a0 msgr2=0x7f277c0fa430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:44.808 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.820+0000 7f278172b700 1 -- 172.21.15.139:0/2026208059 shutdown_connections 2021-11-20T06:03:44.809 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.820+0000 7f278172b700 1 -- 172.21.15.139:0/2026208059 wait complete. 2021-11-20T06:03:44.817 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2021-11-20T06:03:44.944 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.957+0000 7fb37ecfa700 1 -- 172.21.15.139:0/2203568817 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3780f7570 msgr2=0x7fb3780f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:44.945 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.957+0000 7fb37ecfa700 1 --2- 172.21.15.139:0/2203568817 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3780f7570 0x7fb3780f5990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fb364004660 tx=0x7fb364009b30).stop 2021-11-20T06:03:44.945 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.957+0000 7fb37ecfa700 1 -- 172.21.15.139:0/2203568817 shutdown_connections 2021-11-20T06:03:44.945 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.957+0000 7fb37ecfa700 1 --2- 172.21.15.139:0/2203568817 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3780f7570 0x7fb3780f5990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.945 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.957+0000 7fb37ecfa700 1 --2- 172.21.15.139:0/2203568817 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3780f6bd0 0x7fb3780f6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.957+0000 7fb37ecfa700 1 -- 172.21.15.139:0/2203568817 >> 172.21.15.139:0/2203568817 conn(0x7fb3780f04a0 msgr2=0x7fb3780f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:44.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.957+0000 7fb37ecfa700 1 -- 172.21.15.139:0/2203568817 shutdown_connections 2021-11-20T06:03:44.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.957+0000 7fb37ecfa700 1 -- 172.21.15.139:0/2203568817 wait complete. 2021-11-20T06:03:44.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.957+0000 7fb37ecfa700 1 Processor -- start 2021-11-20T06:03:44.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.957+0000 7fb37ecfa700 1 -- start start 2021-11-20T06:03:44.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb37ecfa700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3780f6bd0 0x7fb378106dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:44.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb37ecfa700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3780f7570 0x7fb3781072b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:44.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb37ecfa700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb378107860 con 0x7fb3780f7570 2021-11-20T06:03:44.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb37ecfa700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3781079a0 con 0x7fb3780f6bd0 2021-11-20T06:03:44.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb377fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3780f7570 0x7fb3781072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:44.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb377fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3780f7570 0x7fb3781072b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57540/0 (socket says 172.21.15.139:57540) 2021-11-20T06:03:44.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb377fff700 1 -- 172.21.15.139:0/4186783344 learned_addr learned my addr 172.21.15.139:0/4186783344 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:44.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb37ca96700 1 --2- 172.21.15.139:0/4186783344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3780f6bd0 0x7fb378106dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:44.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb377fff700 1 -- 172.21.15.139:0/4186783344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3780f6bd0 msgr2=0x7fb378106dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:44.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb377fff700 1 --2- 172.21.15.139:0/4186783344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3780f6bd0 0x7fb378106dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:44.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb377fff700 1 -- 172.21.15.139:0/4186783344 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb364005040 con 0x7fb3780f7570 2021-11-20T06:03:44.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb37ca96700 1 --2- 172.21.15.139:0/4186783344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3780f6bd0 0x7fb378106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-11-20T06:03:44.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb377fff700 1 --2- 172.21.15.139:0/4186783344 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3780f7570 0x7fb3781072b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fb364000c00 tx=0x7fb36400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:44.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb3757fa700 1 -- 172.21.15.139:0/4186783344 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb364019070 con 0x7fb3780f7570 2021-11-20T06:03:44.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb3757fa700 1 -- 172.21.15.139:0/4186783344 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb364007e70 con 0x7fb3780f7570 2021-11-20T06:03:44.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb37ecfa700 1 -- 172.21.15.139:0/4186783344 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37810c3e0 con 0x7fb3780f7570 2021-11-20T06:03:44.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb3757fa700 1 -- 172.21.15.139:0/4186783344 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb364014c90 con 0x7fb3780f7570 2021-11-20T06:03:44.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.958+0000 7fb37ecfa700 1 -- 172.21.15.139:0/4186783344 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37810c9a0 con 0x7fb3780f7570 2021-11-20T06:03:44.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.959+0000 7fb37ecfa700 1 -- 172.21.15.139:0/4186783344 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3780fdd60 con 0x7fb3780f7570 2021-11-20T06:03:44.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.960+0000 7fb3757fa700 1 -- 172.21.15.139:0/4186783344 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb364025070 con 0x7fb3780f7570 2021-11-20T06:03:44.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.960+0000 7fb3757fa700 1 --2- 172.21.15.139:0/4186783344 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb36805c2a0 0x7fb36805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:44.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.961+0000 7fb3757fa700 1 -- 172.21.15.139:0/4186783344 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fb364076f20 con 0x7fb3780f7570 2021-11-20T06:03:44.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.961+0000 7fb37ca96700 1 --2- 172.21.15.139:0/4186783344 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb36805c2a0 0x7fb36805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:44.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.961+0000 7fb37ca96700 1 --2- 172.21.15.139:0/4186783344 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb36805c2a0 0x7fb36805e760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fb36c009fd0 tx=0x7fb36c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:44.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:44.964+0000 7fb3757fa700 1 -- 172.21.15.139:0/4186783344 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3640a32b0 con 0x7fb3780f7570 2021-11-20T06:03:45.107 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.122+0000 7fb37ecfa700 1 -- 172.21.15.139:0/4186783344 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fb37810cfe0 con 0x7fb3780f7570 2021-11-20T06:03:45.108 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.123+0000 7fb3757fa700 1 -- 172.21.15.139:0/4186783344 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7fb364049be0 con 0x7fb3780f7570 2021-11-20T06:03:45.109 INFO:teuthology.orchestra.run.smithi139.stdout:pg_num: 32 2021-11-20T06:03:45.110 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.125+0000 7fb37ecfa700 1 -- 172.21.15.139:0/4186783344 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb36805c2a0 msgr2=0x7fb36805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:45.110 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.125+0000 7fb37ecfa700 1 --2- 172.21.15.139:0/4186783344 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb36805c2a0 0x7fb36805e760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fb36c009fd0 tx=0x7fb36c00b040).stop 2021-11-20T06:03:45.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.125+0000 7fb37ecfa700 1 -- 172.21.15.139:0/4186783344 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3780f7570 msgr2=0x7fb3781072b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:45.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.125+0000 7fb37ecfa700 1 --2- 172.21.15.139:0/4186783344 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3780f7570 0x7fb3781072b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fb364000c00 tx=0x7fb36400b040).stop 2021-11-20T06:03:45.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.125+0000 7fb37ecfa700 1 -- 172.21.15.139:0/4186783344 shutdown_connections 2021-11-20T06:03:45.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.125+0000 7fb37ecfa700 1 --2- 172.21.15.139:0/4186783344 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb36805c2a0 0x7fb36805e760 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.125+0000 7fb37ecfa700 1 --2- 172.21.15.139:0/4186783344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3780f6bd0 0x7fb378106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.125+0000 7fb37ecfa700 1 --2- 172.21.15.139:0/4186783344 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3780f7570 0x7fb3781072b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.125+0000 7fb37ecfa700 1 -- 172.21.15.139:0/4186783344 >> 172.21.15.139:0/4186783344 conn(0x7fb3780f04a0 msgr2=0x7fb3780f4860 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:45.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.125+0000 7fb37ecfa700 1 -- 172.21.15.139:0/4186783344 shutdown_connections 2021-11-20T06:03:45.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.125+0000 7fb37ecfa700 1 -- 172.21.15.139:0/4186783344 wait complete. 2021-11-20T06:03:45.122 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-11-20T06:03:45.122 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:03:45.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.266+0000 7fd5906b8700 1 -- 172.21.15.139:0/2609179357 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5880f55a0 msgr2=0x7fd5880f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:45.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.266+0000 7fd5906b8700 1 --2- 172.21.15.139:0/2609179357 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5880f55a0 0x7fd5880f59c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fd578004660 tx=0x7fd578009b30).stop 2021-11-20T06:03:45.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.266+0000 7fd5906b8700 1 -- 172.21.15.139:0/2609179357 shutdown_connections 2021-11-20T06:03:45.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.266+0000 7fd5906b8700 1 --2- 172.21.15.139:0/2609179357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5880f6840 0x7fd5880f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.266+0000 7fd5906b8700 1 --2- 172.21.15.139:0/2609179357 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5880f55a0 0x7fd5880f59c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.266+0000 7fd5906b8700 1 -- 172.21.15.139:0/2609179357 >> 172.21.15.139:0/2609179357 conn(0x7fd5880f0ce0 msgr2=0x7fd5880f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:45.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.266+0000 7fd5906b8700 1 -- 172.21.15.139:0/2609179357 shutdown_connections 2021-11-20T06:03:45.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.266+0000 7fd5906b8700 1 -- 172.21.15.139:0/2609179357 wait complete. 2021-11-20T06:03:45.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.266+0000 7fd5906b8700 1 Processor -- start 2021-11-20T06:03:45.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.266+0000 7fd5906b8700 1 -- start start 2021-11-20T06:03:45.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.266+0000 7fd5906b8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5880f55a0 0x7fd588106e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.266+0000 7fd5906b8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5880f6840 0x7fd588107330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.266+0000 7fd5906b8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5881078e0 con 0x7fd5880f55a0 2021-11-20T06:03:45.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.266+0000 7fd5906b8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd588107a20 con 0x7fd5880f6840 2021-11-20T06:03:45.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.267+0000 7fd58e454700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5880f55a0 0x7fd588106e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:45.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.267+0000 7fd58e454700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5880f55a0 0x7fd588106e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57548/0 (socket says 172.21.15.139:57548) 2021-11-20T06:03:45.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.267+0000 7fd58e454700 1 -- 172.21.15.139:0/2603564879 learned_addr learned my addr 172.21.15.139:0/2603564879 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:45.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.267+0000 7fd58dc53700 1 --2- 172.21.15.139:0/2603564879 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5880f6840 0x7fd588107330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:45.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.267+0000 7fd58e454700 1 -- 172.21.15.139:0/2603564879 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5880f6840 msgr2=0x7fd588107330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:45.262 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.267+0000 7fd58e454700 1 --2- 172.21.15.139:0/2603564879 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5880f6840 0x7fd588107330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.262 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.267+0000 7fd58e454700 1 -- 172.21.15.139:0/2603564879 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd578005040 con 0x7fd5880f55a0 2021-11-20T06:03:45.262 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.267+0000 7fd58dc53700 1 --2- 172.21.15.139:0/2603564879 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5880f6840 0x7fd588107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:03:45.263 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.267+0000 7fd58e454700 1 --2- 172.21.15.139:0/2603564879 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5880f55a0 0x7fd588106e40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fd578004e30 tx=0x7fd578009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:45.263 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.267+0000 7fd586ffd700 1 -- 172.21.15.139:0/2603564879 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd578018070 con 0x7fd5880f55a0 2021-11-20T06:03:45.263 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.267+0000 7fd586ffd700 1 -- 172.21.15.139:0/2603564879 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd578013410 con 0x7fd5880f55a0 2021-11-20T06:03:45.263 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.267+0000 7fd586ffd700 1 -- 172.21.15.139:0/2603564879 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd57801c3c0 con 0x7fd5880f55a0 2021-11-20T06:03:45.263 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.267+0000 7fd5906b8700 1 -- 172.21.15.139:0/2603564879 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd58810c460 con 0x7fd5880f55a0 2021-11-20T06:03:45.264 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.267+0000 7fd5906b8700 1 -- 172.21.15.139:0/2603564879 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58810ca20 con 0x7fd5880f55a0 2021-11-20T06:03:45.264 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.268+0000 7fd5906b8700 1 -- 172.21.15.139:0/2603564879 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd588027a00 con 0x7fd5880f55a0 2021-11-20T06:03:45.264 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.268+0000 7fd586ffd700 1 -- 172.21.15.139:0/2603564879 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd57801d420 con 0x7fd5880f55a0 2021-11-20T06:03:45.264 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.269+0000 7fd586ffd700 1 --2- 172.21.15.139:0/2603564879 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd57005c2a0 0x7fd57005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.264 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.269+0000 7fd586ffd700 1 -- 172.21.15.139:0/2603564879 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fd5780779f0 con 0x7fd5880f55a0 2021-11-20T06:03:45.265 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.269+0000 7fd58dc53700 1 --2- 172.21.15.139:0/2603564879 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd57005c2a0 0x7fd57005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:45.265 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.269+0000 7fd58dc53700 1 --2- 172.21.15.139:0/2603564879 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd57005c2a0 0x7fd57005e760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fd57c009760 tx=0x7fd57c009c60).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:45.265 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.273+0000 7fd586ffd700 1 -- 172.21.15.139:0/2603564879 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd578079e60 con 0x7fd5880f55a0 2021-11-20T06:03:45.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:45 smithi178 conmon[36600]: audit 2021-11-20T06:03:44.200636+0000 mon.smithi139 (mon.0) 530 : audit 2021-11-20T06:03:45.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:45 smithi178 conmon[36600]: [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:45.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:45 smithi178 conmon[36600]: audit 2021-11-20T06:03:44.211832+0000 mon.smithi139 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.139:0/1827783017' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T06:03:45.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:45 smithi178 conmon[36600]: cluster 2021-11-20T06:03:44.211943+0000 mgr.smithi139.dplyxn (mgr.14162) 198 : cluster [DBG] pgmap v159: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2021-11-20T06:03:45.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:45 smithi178 conmon[36600]: audit 2021-11-20T 2021-11-20T06:03:45.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:45 smithi178 conmon[36600]: 06:03:44.515683+0000 mon.smithi139 (mon.0) 532 : audit 2021-11-20T06:03:45.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:45 smithi178 conmon[36600]: [DBG] from='client.? 172.21.15.139:0/1131473791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T06:03:45.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:45 smithi178 conmon[36600]: audit 2021-11-20T06: 2021-11-20T06:03:45.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:45 smithi178 conmon[36600]: 03:44.818858+0000 mon.smithi139 (mon.0) 533 : audit [DBG] 2021-11-20T06:03:45.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:45 smithi178 conmon[36600]: from='client.? 172.21.15.139:0/2026208059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T06:03:45.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:45 smithi178 conmon[36600]: audit 2021-11-20T06:03:45.123763+0000 2021-11-20T06:03:45.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:45 smithi178 conmon[36600]: mon.smithi139 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.139:0/4186783344' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T06:03:45.463 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.477+0000 7fd5906b8700 1 -- 172.21.15.139:0/2603564879 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5880fee30 con 0x7fd5880f55a0 2021-11-20T06:03:45.463 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.477+0000 7fd586ffd700 1 -- 172.21.15.139:0/2603564879 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2945 (secure 0 0 0) 0x7fd57804a730 con 0x7fd5880f55a0 2021-11-20T06:03:45.464 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:03:45.464 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14442,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6829/2967963544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2967963544},{"type":"v1","addr":"172.21.15.139:6829","nonce":2967963544}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":14},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13}],"filesystems":[{"mdsmap":{"epoch":11,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:03:34.486673+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24237},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14426":{"gid":14426,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.139:6827/724620583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":724620583},{"type":"v1","addr":"172.21.15.139:6827","nonce":724620583}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24237":{"gid":24237,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.178:6825/1867304351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1867304351},{"type":"v1","addr":"172.21.15.178:6825","nonce":1867304351}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:03:45.465 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.479+0000 7fd5906b8700 1 -- 172.21.15.139:0/2603564879 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd57005c2a0 msgr2=0x7fd57005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:45.466 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.479+0000 7fd5906b8700 1 --2- 172.21.15.139:0/2603564879 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd57005c2a0 0x7fd57005e760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fd57c009760 tx=0x7fd57c009c60).stop 2021-11-20T06:03:45.466 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.479+0000 7fd5906b8700 1 -- 172.21.15.139:0/2603564879 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5880f55a0 msgr2=0x7fd588106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:45.466 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.479+0000 7fd5906b8700 1 --2- 172.21.15.139:0/2603564879 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5880f55a0 0x7fd588106e40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fd578004e30 tx=0x7fd578009b30).stop 2021-11-20T06:03:45.466 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.479+0000 7fd5906b8700 1 -- 172.21.15.139:0/2603564879 shutdown_connections 2021-11-20T06:03:45.467 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.479+0000 7fd5906b8700 1 --2- 172.21.15.139:0/2603564879 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd57005c2a0 0x7fd57005e760 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.467 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.479+0000 7fd5906b8700 1 --2- 172.21.15.139:0/2603564879 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd5880f55a0 0x7fd588106e40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.467 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.479+0000 7fd5906b8700 1 --2- 172.21.15.139:0/2603564879 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5880f6840 0x7fd588107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.467 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.479+0000 7fd5906b8700 1 -- 172.21.15.139:0/2603564879 >> 172.21.15.139:0/2603564879 conn(0x7fd5880f0ce0 msgr2=0x7fd5880fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:45.468 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.479+0000 7fd5906b8700 1 -- 172.21.15.139:0/2603564879 shutdown_connections 2021-11-20T06:03:45.468 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.479+0000 7fd5906b8700 1 -- 172.21.15.139:0/2603564879 wait complete. 2021-11-20T06:03:45.468 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 14 2021-11-20T06:03:45.475 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:45 smithi139 conmon[33501]: audit 2021-11-20T06:03:44.200636+0000 2021-11-20T06:03:45.476 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:45 smithi139 conmon[33501]: mon.smithi139 (mon.0) 530 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:45.476 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:45 smithi139 conmon[33501]: audit 2021-11-20T06:03:44.211832+0000 mon.smithi139 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.139:0/1827783017' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T06:03:45.476 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:45 smithi139 conmon[33501]: cluster 2021-11-20T06:03:44.211943+0000 mgr.smithi139.dplyxn (mgr.14162) 198 : cluster [DBG] pgmap v159: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2021-11-20T06:03:45.476 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:45 smithi139 conmon[33501]: audit 2021-11-20T06:03:44.515683+0000 mon.smithi139 (mon.0) 532 : audit [DBG] from='client.? 172.21.15.139:0/1131473791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T06:03:45.477 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:45 smithi139 conmon[33501]: audit 2021-11-20T06:03:44.818858+0000 mon.smithi139 (mon.0) 533 : audit [DBG] from='client.? 172.21.15.139:0/2026208059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T06:03:45.477 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:45 smithi139 conmon[33501]: audit 2021-11-20T06:03:45.123763+0000 mon.smithi139 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.139:0/4186783344' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T06:03:45.478 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 11, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:03:34.486673+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24237}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14426': {'gid': 14426, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.139:6827/724620583', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 724620583}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 724620583}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24237': {'gid': 24237, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.178:6825/1867304351', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 1867304351}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 1867304351}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:03:45.479 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-11-20T06:03:45.479 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.cephfs.smithi178.blvkbx status 2021-11-20T06:03:45.598 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.610+0000 7f8db89ab700 1 -- 172.21.15.139:0/1493468311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db40f5140 msgr2=0x7f8db40f78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:45.600 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.610+0000 7f8db89ab700 1 --2- 172.21.15.139:0/1493468311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db40f5140 0x7f8db40f78c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f8da80099e0 tx=0x7f8da8004d10).stop 2021-11-20T06:03:45.600 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.610+0000 7f8db89ab700 1 -- 172.21.15.139:0/1493468311 shutdown_connections 2021-11-20T06:03:45.600 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.610+0000 7f8db89ab700 1 --2- 172.21.15.139:0/1493468311 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db40f5140 0x7f8db40f78c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.600 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.610+0000 7f8db89ab700 1 --2- 172.21.15.139:0/1493468311 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8db40f47a0 0x7f8db40f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.601 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.610+0000 7f8db89ab700 1 -- 172.21.15.139:0/1493468311 >> 172.21.15.139:0/1493468311 conn(0x7f8db40f02c0 msgr2=0x7f8db40f26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:45.601 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.610+0000 7f8db89ab700 1 -- 172.21.15.139:0/1493468311 shutdown_connections 2021-11-20T06:03:45.601 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.610+0000 7f8db89ab700 1 -- 172.21.15.139:0/1493468311 wait complete. 2021-11-20T06:03:45.601 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.611+0000 7f8db89ab700 1 Processor -- start 2021-11-20T06:03:45.601 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.611+0000 7f8db89ab700 1 -- start start 2021-11-20T06:03:45.602 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.611+0000 7f8db89ab700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db40f47a0 0x7f8db4106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.602 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.611+0000 7f8db89ab700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8db40f5140 0x7f8db41073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.602 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.611+0000 7f8db89ab700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db4107990 con 0x7f8db40f47a0 2021-11-20T06:03:45.603 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.611+0000 7f8db89ab700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db4107ad0 con 0x7f8db40f5140 2021-11-20T06:03:45.603 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.611+0000 7f8db37fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db40f47a0 0x7f8db4106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:45.603 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.611+0000 7f8db37fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db40f47a0 0x7f8db4106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57556/0 (socket says 172.21.15.139:57556) 2021-11-20T06:03:45.604 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.611+0000 7f8db37fe700 1 -- 172.21.15.139:0/2216181296 learned_addr learned my addr 172.21.15.139:0/2216181296 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:45.605 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.611+0000 7f8db2ffd700 1 --2- 172.21.15.139:0/2216181296 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8db40f5140 0x7f8db41073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:45.605 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.611+0000 7f8db37fe700 1 -- 172.21.15.139:0/2216181296 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8db40f5140 msgr2=0x7f8db41073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:45.605 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.611+0000 7f8db37fe700 1 --2- 172.21.15.139:0/2216181296 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8db40f5140 0x7f8db41073e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.606 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.611+0000 7f8db37fe700 1 -- 172.21.15.139:0/2216181296 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da8009640 con 0x7f8db40f47a0 2021-11-20T06:03:45.606 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.611+0000 7f8db37fe700 1 --2- 172.21.15.139:0/2216181296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db40f47a0 0x7f8db4106ef0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f8da400c530 tx=0x7f8da4007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:45.607 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.612+0000 7f8d9bfff700 1 -- 172.21.15.139:0/2216181296 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8da400f650 con 0x7f8db40f47a0 2021-11-20T06:03:45.607 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.612+0000 7f8db89ab700 1 -- 172.21.15.139:0/2216181296 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db4108470 con 0x7f8db40f47a0 2021-11-20T06:03:45.607 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.612+0000 7f8d9bfff700 1 -- 172.21.15.139:0/2216181296 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8da400fc60 con 0x7f8db40f47a0 2021-11-20T06:03:45.608 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.612+0000 7f8d9bfff700 1 -- 172.21.15.139:0/2216181296 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8da4012d20 con 0x7f8db40f47a0 2021-11-20T06:03:45.608 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.612+0000 7f8db89ab700 1 -- 172.21.15.139:0/2216181296 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db41089b0 con 0x7f8db40f47a0 2021-11-20T06:03:45.608 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.613+0000 7f8d9bfff700 1 -- 172.21.15.139:0/2216181296 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8da4019030 con 0x7f8db40f47a0 2021-11-20T06:03:45.608 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.613+0000 7f8d9bfff700 1 --2- 172.21.15.139:0/2216181296 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8d9c05c2b0 0x7f8d9c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.609 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.613+0000 7f8d9bfff700 1 -- 172.21.15.139:0/2216181296 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f8da4075b30 con 0x7f8db40f47a0 2021-11-20T06:03:45.609 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.613+0000 7f8db2ffd700 1 --2- 172.21.15.139:0/2216181296 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8d9c05c2b0 0x7f8d9c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:45.609 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.614+0000 7f8db2ffd700 1 --2- 172.21.15.139:0/2216181296 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8d9c05c2b0 0x7f8d9c05e770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f8da8000f80 tx=0x7f8da8004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:45.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.615+0000 7f8dcb7b8b80 1 Processor -- start 2021-11-20T06:03:45.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.615+0000 7f8dcb7b8b80 1 -- start start 2021-11-20T06:03:45.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.615+0000 7f8dcb7b8b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 0x55f229bd3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.615+0000 7f8dcb7b8b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bd4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.617 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.615+0000 7f8dcb7b8b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f229b124d0 con 0x55f229bd41b0 2021-11-20T06:03:45.617 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.615+0000 7f8dcb7b8b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f229b263e0 con 0x55f229bd38a0 2021-11-20T06:03:45.617 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.615+0000 7f8db2ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bd4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:45.618 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.615+0000 7f8db2ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bd4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57562/0 (socket says 172.21.15.139:57562) 2021-11-20T06:03:45.618 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.615+0000 7f8db2ffd700 1 -- 172.21.15.139:0/2686445573 learned_addr learned my addr 172.21.15.139:0/2686445573 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:45.618 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.616+0000 7f8db2ffd700 1 -- 172.21.15.139:0/2686445573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 msgr2=0x55f229bd3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T06:03:45.618 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.616+0000 7f8db2ffd700 1 --2- 172.21.15.139:0/2686445573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 0x55f229bd3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:03:45.618 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.616+0000 7f8db2ffd700 1 -- 172.21.15.139:0/2686445573 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f229ba23c0 con 0x55f229bd41b0 2021-11-20T06:03:45.619 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.616+0000 7f8db2ffd700 1 --2- 172.21.15.139:0/2686445573 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bd4b30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f8da800b380 tx=0x7f8da801b000).ready entity=mon.0 client_cookie=e1f0657cd36fc561 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:45.619 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.616+0000 7f8d997fa700 1 -- 172.21.15.139:0/2686445573 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8da8019070 con 0x55f229bd41b0 2021-11-20T06:03:45.619 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.616+0000 7f8d997fa700 1 -- 172.21.15.139:0/2686445573 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8da8024070 con 0x55f229bd41b0 2021-11-20T06:03:45.619 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.616+0000 7f8d997fa700 1 -- 172.21.15.139:0/2686445573 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8da801dd90 con 0x55f229bd41b0 2021-11-20T06:03:45.620 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.617+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/2686445573 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 msgr2=0x55f229bd4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:45.620 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.617+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/2686445573 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bd4b30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f8da800b380 tx=0x7f8da801b000).stop 2021-11-20T06:03:45.620 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.617+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/2686445573 shutdown_connections 2021-11-20T06:03:45.621 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.617+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/2686445573 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bd4b30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.621 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.617+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/2686445573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 0x55f229bd3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:03:45.621 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.617+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/2686445573 >> 172.21.15.139:0/2686445573 conn(0x55f229a411d0 msgr2=0x55f229bcee00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:45.621 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.617+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/2686445573 shutdown_connections 2021-11-20T06:03:45.622 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.617+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/2686445573 wait complete. 2021-11-20T06:03:45.622 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.617+0000 7f8dcb7b8b80 1 -- start start 2021-11-20T06:03:45.622 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.617+0000 7f8dcb7b8b80 1 Processor -- start 2021-11-20T06:03:45.622 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.617+0000 7f8dcb7b8b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 0x55f229bdcbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.622 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.617+0000 7f8dcb7b8b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bdd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.623 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.617+0000 7f8dcb7b8b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f229b26520 con 0x55f229bd41b0 2021-11-20T06:03:45.623 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.617+0000 7f8dcb7b8b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f229b26aa0 con 0x55f229bd38a0 2021-11-20T06:03:45.623 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8dcb7b8b80 10 client.0 ll_register_callbacks cb 0x55f229b938b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T06:03:45.623 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8db2ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bdd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:45.624 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8db2ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bdd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57566/0 (socket says 172.21.15.139:57566) 2021-11-20T06:03:45.624 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8db2ffd700 1 -- 172.21.15.139:0/4013219679 learned_addr learned my addr 172.21.15.139:0/4013219679 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:45.624 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8db2ffd700 1 -- 172.21.15.139:0/4013219679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 msgr2=0x55f229bdcbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T06:03:45.624 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8db37fe700 1 --2- 172.21.15.139:0/4013219679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 0x55f229bdcbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:45.624 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8db2ffd700 1 --2- 172.21.15.139:0/4013219679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 0x55f229bdcbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8db2ffd700 1 -- 172.21.15.139:0/4013219679 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db40f8160 con 0x55f229bd41b0 2021-11-20T06:03:45.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8db2ffd700 1 --2- 172.21.15.139:0/4013219679 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bdd0a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f8da8004a60 tx=0x7f8da8022040).ready entity=mon.0 client_cookie=4c4b137878ae716 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:45.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8d927fc700 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2021-11-20T06:03:45.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8d927fc700 1 -- 172.21.15.139:0/4013219679 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8da8019070 con 0x55f229bd41b0 2021-11-20T06:03:45.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8d927fc700 1 -- 172.21.15.139:0/4013219679 <== mon.0 v2:172.21.15.139:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f8da8007980 con 0x55f229bd41b0 2021-11-20T06:03:45.626 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/4013219679 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f229ba1ce0 con 0x55f229bd41b0 2021-11-20T06:03:45.626 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8d927fc700 1 -- 172.21.15.139:0/4013219679 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8da8007be0 con 0x55f229bd41b0 2021-11-20T06:03:45.626 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8dcb7b8b80 10 client.14502 fetch_fsmap learned FSMap version 14 2021-11-20T06:03:45.626 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/4013219679 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=3+}) v3 -- 0x55f229b15630 con 0x55f229bd41b0 2021-11-20T06:03:45.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8d927fc700 1 -- 172.21.15.139:0/4013219679 <== mon.0 v2:172.21.15.139:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x55f229ba1ce0 con 0x55f229bd41b0 2021-11-20T06:03:45.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.618+0000 7f8d927fc700 1 -- 172.21.15.139:0/4013219679 <== mon.0 v2:172.21.15.139:3300/0 5 ==== fsmap(e 14) v1 ==== 1712+0+0 (secure 0 0 0) 0x7f8da8013b40 con 0x55f229bd41b0 2021-11-20T06:03:45.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.619+0000 7f8dcb7b8b80 10 client.14502 fetch_fsmap finished waiting for FSMap version 14 2021-11-20T06:03:45.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.619+0000 7f8dcb7b8b80 10 client.14502 resolve_mds: resolved name 'cephfs.smithi178.blvkbx' to daemon mds.cephfs.smithi178.blvkbx 2021-11-20T06:03:45.628 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.619+0000 7f8dcb7b8b80 20 client.14502 populate_metadata read hostname 'smithi139' 2021-11-20T06:03:45.628 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.619+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/4013219679 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55f229be58a0 0x55f229be7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.619+0000 7f8dcb7b8b80 4 client.14502 mds_command: new command op to 24237 tid=0[{"prefix": "get_command_descriptions"}] 2021-11-20T06:03:45.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.619+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/4013219679 --> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f229b1cdf0 con 0x55f229be58a0 2021-11-20T06:03:45.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.619+0000 7f8db37fe700 1 --2- 172.21.15.139:0/4013219679 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55f229be58a0 0x55f229be7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:45.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.619+0000 7f8d927fc700 10 client.14502 ms_handle_connect on v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:45.630 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.619+0000 7f8db37fe700 1 --2- 172.21.15.139:0/4013219679 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55f229be58a0 0x55f229be7c90 crc :-1 s=READY pgs=9 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=4ad5fa782107220c server_cookie=39738b013b88b12a in_seq=0 out_seq=0 2021-11-20T06:03:45.630 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.620+0000 7f8d927fc700 1 -- 172.21.15.139:0/4013219679 <== mds.0 v2:172.21.15.178:6824/1867304351 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x55f229b1cdf0 con 0x55f229be58a0 2021-11-20T06:03:45.630 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.620+0000 7f8d927fc700 10 client.14502 handle_command_reply: tid=0 2021-11-20T06:03:45.630 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.620+0000 7f8dcb7b8b80 1 client.14502 shutdown 2021-11-20T06:03:45.631 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.620+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/4013219679 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 msgr2=0x55f229bdd0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:45.631 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.620+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/4013219679 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bdd0a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f8da8004a60 tx=0x7f8da8022040).stop 2021-11-20T06:03:45.631 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.620+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/4013219679 shutdown_connections 2021-11-20T06:03:45.631 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.620+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/4013219679 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55f229be58a0 0x55f229be7c90 crc :-1 s=READY pgs=9 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.631 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.620+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/4013219679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 0x55f229bdcbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.632 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.620+0000 7f8d927fc700 0 client.14502 ms_handle_reset on v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:45.632 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.620+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/4013219679 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bdd0a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.632 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.620+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/4013219679 >> 172.21.15.139:0/4013219679 conn(0x55f229a411d0 msgr2=0x55f229bce9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:45.632 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.620+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/4013219679 shutdown_connections 2021-11-20T06:03:45.633 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.620+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/4013219679 wait complete. 2021-11-20T06:03:45.633 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.620+0000 7f8dcb7b8b80 20 client.14502 trim_cache size 0 max 16384 2021-11-20T06:03:45.639 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.636+0000 7f8dcb7b8b80 1 Processor -- start 2021-11-20T06:03:45.639 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.636+0000 7f8dcb7b8b80 1 -- start start 2021-11-20T06:03:45.639 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.637+0000 7f8dcb7b8b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 0x55f229bcfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.640 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.637+0000 7f8dcb7b8b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bd0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.640 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.637+0000 7f8dcb7b8b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f229af44a0 con 0x55f229bd41b0 2021-11-20T06:03:45.640 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.637+0000 7f8dcb7b8b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f229bd45d0 con 0x55f229bd38a0 2021-11-20T06:03:45.640 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.637+0000 7f8db2ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bd0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:45.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.637+0000 7f8db2ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bd0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57572/0 (socket says 172.21.15.139:57572) 2021-11-20T06:03:45.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.637+0000 7f8db2ffd700 1 -- 172.21.15.139:0/1171238122 learned_addr learned my addr 172.21.15.139:0/1171238122 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:45.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.637+0000 7f8db37fe700 1 --2- 172.21.15.139:0/1171238122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 0x55f229bcfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:45.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.637+0000 7f8db2ffd700 1 -- 172.21.15.139:0/1171238122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 msgr2=0x55f229bcfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T06:03:45.642 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.637+0000 7f8db2ffd700 1 --2- 172.21.15.139:0/1171238122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 0x55f229bcfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.642 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.637+0000 7f8db2ffd700 1 -- 172.21.15.139:0/1171238122 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f229bdc630 con 0x55f229bd41b0 2021-11-20T06:03:45.642 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.637+0000 7f8db2ffd700 1 --2- 172.21.15.139:0/1171238122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bd0300 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f8da8004fd0 tx=0x7f8da8022040).ready entity=mon.0 client_cookie=40ef6be0c96a5999 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:45.642 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.637+0000 7f8d99ffb700 1 -- 172.21.15.139:0/1171238122 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8da8019070 con 0x55f229bd41b0 2021-11-20T06:03:45.642 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.637+0000 7f8d99ffb700 1 -- 172.21.15.139:0/1171238122 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8da802e070 con 0x55f229bd41b0 2021-11-20T06:03:45.643 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.638+0000 7f8d99ffb700 1 -- 172.21.15.139:0/1171238122 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8da802aa90 con 0x55f229bd41b0 2021-11-20T06:03:45.643 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.638+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/1171238122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 msgr2=0x55f229bd0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:45.643 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.638+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/1171238122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bd0300 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f8da8004fd0 tx=0x7f8da8022040).stop 2021-11-20T06:03:45.643 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/1171238122 shutdown_connections 2021-11-20T06:03:45.644 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/1171238122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229bd0300 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.644 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/1171238122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 0x55f229bcfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.644 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/1171238122 >> 172.21.15.139:0/1171238122 conn(0x55f229a411d0 msgr2=0x55f229bcca20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:45.644 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/1171238122 shutdown_connections 2021-11-20T06:03:45.644 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/1171238122 wait complete. 2021-11-20T06:03:45.645 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8dcb7b8b80 1 -- start start 2021-11-20T06:03:45.645 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8dcb7b8b80 1 Processor -- start 2021-11-20T06:03:45.645 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8dcb7b8b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 0x55f229c32230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.645 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8dcb7b8b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229c32720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.646 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8dcb7b8b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f229c32cd0 con 0x55f229bd41b0 2021-11-20T06:03:45.646 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8dcb7b8b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f229c32e10 con 0x55f229bd38a0 2021-11-20T06:03:45.646 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8dcb7b8b80 10 client.0 ll_register_callbacks cb 0x55f229b938b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T06:03:45.646 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8db2ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229c32720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:45.647 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8db2ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229c32720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57574/0 (socket says 172.21.15.139:57574) 2021-11-20T06:03:45.647 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.639+0000 7f8db2ffd700 1 -- 172.21.15.139:0/1541993008 learned_addr learned my addr 172.21.15.139:0/1541993008 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:45.647 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8db37fe700 1 --2- 172.21.15.139:0/1541993008 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 0x55f229c32230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:45.647 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8db2ffd700 1 -- 172.21.15.139:0/1541993008 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 msgr2=0x55f229c32230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T06:03:45.647 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8db2ffd700 1 --2- 172.21.15.139:0/1541993008 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 0x55f229c32230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.648 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8db2ffd700 1 -- 172.21.15.139:0/1541993008 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f229b15630 con 0x55f229bd41b0 2021-11-20T06:03:45.648 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8db2ffd700 1 --2- 172.21.15.139:0/1541993008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229c32720 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f8da8004710 tx=0x7f8da8022040).ready entity=mon.0 client_cookie=1d098f1f4900dd0b server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:45.648 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8d927fc700 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2021-11-20T06:03:45.648 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8d927fc700 1 -- 172.21.15.139:0/1541993008 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8da8019070 con 0x55f229bd41b0 2021-11-20T06:03:45.649 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8d927fc700 1 -- 172.21.15.139:0/1541993008 <== mon.0 v2:172.21.15.139:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f8da802aa20 con 0x55f229bd41b0 2021-11-20T06:03:45.649 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/1541993008 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f229bcdfc0 con 0x55f229bd41b0 2021-11-20T06:03:45.649 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8d927fc700 1 -- 172.21.15.139:0/1541993008 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8da802abe0 con 0x55f229bd41b0 2021-11-20T06:03:45.649 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8dcb7b8b80 10 client.14506 fetch_fsmap learned FSMap version 14 2021-11-20T06:03:45.649 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/1541993008 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=3+}) v3 -- 0x7f8db410f7e0 con 0x55f229bd41b0 2021-11-20T06:03:45.650 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8d927fc700 1 -- 172.21.15.139:0/1541993008 <== mon.0 v2:172.21.15.139:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x55f229bcdfc0 con 0x55f229bd41b0 2021-11-20T06:03:45.650 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8d927fc700 1 -- 172.21.15.139:0/1541993008 <== mon.0 v2:172.21.15.139:3300/0 5 ==== fsmap(e 14) v1 ==== 1712+0+0 (secure 0 0 0) 0x7f8da8020c60 con 0x55f229bd41b0 2021-11-20T06:03:45.650 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8dcb7b8b80 10 client.14506 fetch_fsmap finished waiting for FSMap version 14 2021-11-20T06:03:45.650 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8dcb7b8b80 10 client.14506 resolve_mds: resolved name 'cephfs.smithi178.blvkbx' to daemon mds.cephfs.smithi178.blvkbx 2021-11-20T06:03:45.651 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8dcb7b8b80 20 client.14506 populate_metadata read hostname 'smithi139' 2021-11-20T06:03:45.651 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/1541993008 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55f229c384d0 0x55f229c3a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:45.651 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8dcb7b8b80 4 client.14506 mds_command: new command op to 24237 tid=0[{"prefix": "status"}] 2021-11-20T06:03:45.651 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.640+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/1541993008 --> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f229b140c0 con 0x55f229c384d0 2021-11-20T06:03:45.652 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8db37fe700 1 --2- 172.21.15.139:0/1541993008 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55f229c384d0 0x55f229c3a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:45.652 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8d927fc700 10 client.14506 ms_handle_connect on v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:45.652 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8db37fe700 1 --2- 172.21.15.139:0/1541993008 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55f229c384d0 0x55f229c3a8c0 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=f7ef26318bb02f43 server_cookie=ce891c1f5d09a3f3 in_seq=0 out_seq=0 2021-11-20T06:03:45.652 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8d927fc700 1 -- 172.21.15.139:0/1541993008 <== mds.0 v2:172.21.15.178:6824/1867304351 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55f229b140c0 con 0x55f229c384d0 2021-11-20T06:03:45.652 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8d927fc700 10 client.14506 handle_command_reply: tid=0 2021-11-20T06:03:45.653 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8dcb7b8b80 1 client.14506 shutdown 2021-11-20T06:03:45.653 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/1541993008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 msgr2=0x55f229c32720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:45.653 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/1541993008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229c32720 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f8da8004710 tx=0x7f8da8022040).stop 2021-11-20T06:03:45.653 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/1541993008 shutdown_connections 2021-11-20T06:03:45.653 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/1541993008 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55f229c384d0 0x55f229c3a8c0 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.654 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/1541993008 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f229bd38a0 0x55f229c32230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.654 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8dcb7b8b80 1 --2- 172.21.15.139:0/1541993008 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55f229bd41b0 0x55f229c32720 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.654 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/1541993008 >> 172.21.15.139:0/1541993008 conn(0x55f229a411d0 msgr2=0x55f229bcfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:45.654 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8d927fc700 0 client.14506 ms_handle_reset on v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:45.655 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/1541993008 shutdown_connections 2021-11-20T06:03:45.655 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.641+0000 7f8dcb7b8b80 1 -- 172.21.15.139:0/1541993008 wait complete. 2021-11-20T06:03:45.655 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.642+0000 7f8dcb7b8b80 20 client.14506 trim_cache size 0 max 16384 2021-11-20T06:03:45.655 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2021-11-20T06:03:45.656 INFO:teuthology.orchestra.run.smithi139.stdout: "cluster_fsid": "d819f2cc-49c6-11ec-8c2c-001a4aab830c", 2021-11-20T06:03:45.656 INFO:teuthology.orchestra.run.smithi139.stdout: "whoami": 0, 2021-11-20T06:03:45.656 INFO:teuthology.orchestra.run.smithi139.stdout: "id": 24237, 2021-11-20T06:03:45.656 INFO:teuthology.orchestra.run.smithi139.stdout: "want_state": "up:active", 2021-11-20T06:03:45.657 INFO:teuthology.orchestra.run.smithi139.stdout: "state": "up:active", 2021-11-20T06:03:45.657 INFO:teuthology.orchestra.run.smithi139.stdout: "fs_name": "cephfs", 2021-11-20T06:03:45.657 INFO:teuthology.orchestra.run.smithi139.stdout: "rank_uptime": 17.417127972999999, 2021-11-20T06:03:45.657 INFO:teuthology.orchestra.run.smithi139.stdout: "mdsmap_epoch": 11, 2021-11-20T06:03:45.658 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_epoch": 41, 2021-11-20T06:03:45.658 INFO:teuthology.orchestra.run.smithi139.stdout: "osdmap_epoch_barrier": 41, 2021-11-20T06:03:45.658 INFO:teuthology.orchestra.run.smithi139.stdout: "uptime": 17.818633218999999 2021-11-20T06:03:45.658 INFO:teuthology.orchestra.run.smithi139.stdout:} 2021-11-20T06:03:45.660 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.642+0000 7f8db89ab700 1 -- 172.21.15.139:0/2216181296 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8d9c05c2b0 msgr2=0x7f8d9c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:45.660 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.642+0000 7f8db89ab700 1 --2- 172.21.15.139:0/2216181296 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8d9c05c2b0 0x7f8d9c05e770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f8da8000f80 tx=0x7f8da8004d10).stop 2021-11-20T06:03:45.660 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.642+0000 7f8db89ab700 1 -- 172.21.15.139:0/2216181296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db40f47a0 msgr2=0x7f8db4106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:45.661 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.642+0000 7f8db89ab700 1 --2- 172.21.15.139:0/2216181296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db40f47a0 0x7f8db4106ef0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f8da400c530 tx=0x7f8da4007e10).stop 2021-11-20T06:03:45.661 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.642+0000 7f8db89ab700 1 -- 172.21.15.139:0/2216181296 shutdown_connections 2021-11-20T06:03:45.661 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.642+0000 7f8db89ab700 1 --2- 172.21.15.139:0/2216181296 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8d9c05c2b0 0x7f8d9c05e770 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.661 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.642+0000 7f8db89ab700 1 --2- 172.21.15.139:0/2216181296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8db40f47a0 0x7f8db4106ef0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.662 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.642+0000 7f8db89ab700 1 --2- 172.21.15.139:0/2216181296 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8db40f5140 0x7f8db41073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:45.662 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.642+0000 7f8db89ab700 1 -- 172.21.15.139:0/2216181296 >> 172.21.15.139:0/2216181296 conn(0x7f8db40f02c0 msgr2=0x7f8db40fb310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:45.662 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.642+0000 7f8db89ab700 1 -- 172.21.15.139:0/2216181296 shutdown_connections 2021-11-20T06:03:45.662 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:45.642+0000 7f8db89ab700 1 -- 172.21.15.139:0/2216181296 wait complete. 2021-11-20T06:03:45.663 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2021-11-20T06:03:45.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:45.663 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p /home/ubuntu/cephtest/mnt.0 2021-11-20T06:03:45.679 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-11-20T06:03:45.680 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2021-11-20T06:03:45.771 INFO:teuthology.orchestra.run.smithi139.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2021-11-20T06:03:45.772 INFO:teuthology.orchestra.run.smithi139.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2021-11-20T06:03:45.875 INFO:teuthology.orchestra.run.smithi139.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2021-11-20T06:03:45.876 INFO:teuthology.orchestra.run.smithi139.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2021-11-20T06:03:45.879 INFO:tasks.cephfs.kernel_mount:mount command passed 2021-11-20T06:03:45.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:45.880 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2021-11-20T06:03:45.912 INFO:tasks.kclient:client.1 config is {} 2021-11-20T06:03:45.912 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2021-11-20T06:03:45.912 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi178.front.sepia.ceph.com 2021-11-20T06:03:45.913 INFO:tasks.cephfs.mount:self.client.name = client.1 2021-11-20T06:03:45.913 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2021-11-20T06:03:45.913 INFO:tasks.cephfs.mount:self.cephfs_name = None 2021-11-20T06:03:45.914 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2021-11-20T06:03:45.914 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2021-11-20T06:03:45.914 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2021-11-20T06:03:45.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:45.915 DEBUG:teuthology.orchestra.run.smithi178:> ip addr 2021-11-20T06:03:45.935 INFO:teuthology.orchestra.run.smithi178.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2021-11-20T06:03:45.936 INFO:teuthology.orchestra.run.smithi178.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2021-11-20T06:03:45.936 INFO:teuthology.orchestra.run.smithi178.stdout: inet 127.0.0.1/8 scope host lo 2021-11-20T06:03:45.936 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2021-11-20T06:03:45.936 INFO:teuthology.orchestra.run.smithi178.stdout: inet6 ::1/128 scope host 2021-11-20T06:03:45.937 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2021-11-20T06:03:45.937 INFO:teuthology.orchestra.run.smithi178.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T06:03:45.937 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether 0c:c4:7a:d9:3b:20 brd ff:ff:ff:ff:ff:ff 2021-11-20T06:03:45.937 INFO:teuthology.orchestra.run.smithi178.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T06:03:45.938 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether 0c:c4:7a:d9:3b:21 brd ff:ff:ff:ff:ff:ff 2021-11-20T06:03:45.938 INFO:teuthology.orchestra.run.smithi178.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-20T06:03:45.938 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether 0c:c4:7a:88:7e:f8 brd ff:ff:ff:ff:ff:ff 2021-11-20T06:03:45.938 INFO:teuthology.orchestra.run.smithi178.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2021-11-20T06:03:45.938 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether 0c:c4:7a:88:7e:f9 brd ff:ff:ff:ff:ff:ff 2021-11-20T06:03:45.939 INFO:teuthology.orchestra.run.smithi178.stdout: inet 172.21.15.178/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2021-11-20T06:03:45.939 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft 42280sec preferred_lft 42280sec 2021-11-20T06:03:45.939 INFO:teuthology.orchestra.run.smithi178.stdout: inet6 fe80::ec4:7aff:fe88:7ef9/64 scope link noprefixroute 2021-11-20T06:03:45.939 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2021-11-20T06:03:45.940 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2021-11-20T06:03:45.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:45.941 DEBUG:teuthology.orchestra.run.smithi178:> (cd / && exec bash -c ' 2021-11-20T06:03:45.941 DEBUG:teuthology.orchestra.run.smithi178:> set -e 2021-11-20T06:03:45.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link add name ceph-brx type bridge 2021-11-20T06:03:45.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip addr flush dev ceph-brx 2021-11-20T06:03:45.942 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link set ceph-brx up 2021-11-20T06:03:45.942 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2021-11-20T06:03:45.942 DEBUG:teuthology.orchestra.run.smithi178:> ') 2021-11-20T06:03:46.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:46.113 DEBUG:teuthology.orchestra.run.smithi178:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2021-11-20T06:03:46.140 INFO:teuthology.orchestra.run.smithi178.stdout:1 2021-11-20T06:03:46.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:46.142 DEBUG:teuthology.orchestra.run.smithi178:> route 2021-11-20T06:03:46.212 INFO:teuthology.orchestra.run.smithi178.stdout:Kernel IP routing table 2021-11-20T06:03:46.213 INFO:teuthology.orchestra.run.smithi178.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2021-11-20T06:03:46.213 INFO:teuthology.orchestra.run.smithi178.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2021-11-20T06:03:46.213 INFO:teuthology.orchestra.run.smithi178.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2021-11-20T06:03:46.214 INFO:teuthology.orchestra.run.smithi178.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2021-11-20T06:03:46.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:46.214 DEBUG:teuthology.orchestra.run.smithi178:> (cd / && exec bash -c ' 2021-11-20T06:03:46.215 DEBUG:teuthology.orchestra.run.smithi178:> set -e 2021-11-20T06:03:46.215 DEBUG:teuthology.orchestra.run.smithi178:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2021-11-20T06:03:46.216 DEBUG:teuthology.orchestra.run.smithi178:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2021-11-20T06:03:46.216 DEBUG:teuthology.orchestra.run.smithi178:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2021-11-20T06:03:46.216 DEBUG:teuthology.orchestra.run.smithi178:> ') 2021-11-20T06:03:46.289 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:46 smithi178 conmon[36600]: audit 2021-11-20T06:03:45.478067+0000 mon.smithi139 (mon.0) 535 : audit [DBG] 2021-11-20T06:03:46.289 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:46 smithi178 conmon[36600]: from='client.? 172.21.15.139:0/2603564879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:03:46.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:46.497 DEBUG:teuthology.orchestra.run.smithi178:> ip netns list 2021-11-20T06:03:46.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:46 smithi139 conmon[33501]: audit 2021-11-20T06:03:45.478067+0000 mon.smithi139 (mon.0) 535 : audit [DBG] from='client.? 172.21.15.139:0/2603564879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:03:46.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:46.552 DEBUG:teuthology.orchestra.run.smithi178:> ip netns list-id 2021-11-20T06:03:46.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:46.608 DEBUG:teuthology.orchestra.run.smithi178:> (cd / && exec bash -c ' 2021-11-20T06:03:46.608 DEBUG:teuthology.orchestra.run.smithi178:> set -e 2021-11-20T06:03:46.608 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2021-11-20T06:03:46.609 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2021-11-20T06:03:46.609 DEBUG:teuthology.orchestra.run.smithi178:> ') 2021-11-20T06:03:46.700 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2021-11-20T06:03:46.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:46.701 DEBUG:teuthology.orchestra.run.smithi178:> (cd / && exec bash -c ' 2021-11-20T06:03:46.701 DEBUG:teuthology.orchestra.run.smithi178:> set -e 2021-11-20T06:03:46.702 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2021-11-20T06:03:46.702 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2021-11-20T06:03:46.702 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2021-11-20T06:03:46.703 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2021-11-20T06:03:46.703 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2021-11-20T06:03:46.703 DEBUG:teuthology.orchestra.run.smithi178:> ') 2021-11-20T06:03:46.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-20T06:03:46.931 DEBUG:teuthology.orchestra.run.smithi178:> (cd / && exec bash -c ' 2021-11-20T06:03:46.931 DEBUG:teuthology.orchestra.run.smithi178:> set -e 2021-11-20T06:03:46.931 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link set brx.0 up 2021-11-20T06:03:46.932 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link set dev brx.0 master ceph-brx 2021-11-20T06:03:46.932 DEBUG:teuthology.orchestra.run.smithi178:> ') 2021-11-20T06:03:47.045 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-11-20T06:03:47.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.189+0000 7eff6cf84700 1 -- 172.21.15.139:0/239531329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680f6840 msgr2=0x7eff680f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:47.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.189+0000 7eff6cf84700 1 --2- 172.21.15.139:0/239531329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680f6840 0x7eff680f6ca0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7eff50004660 tx=0x7eff50009b30).stop 2021-11-20T06:03:47.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.189+0000 7eff6cf84700 1 -- 172.21.15.139:0/239531329 shutdown_connections 2021-11-20T06:03:47.179 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.189+0000 7eff6cf84700 1 --2- 172.21.15.139:0/239531329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680f6840 0x7eff680f6ca0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.179 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.189+0000 7eff6cf84700 1 --2- 172.21.15.139:0/239531329 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff680f55a0 0x7eff680f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.179 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.189+0000 7eff6cf84700 1 -- 172.21.15.139:0/239531329 >> 172.21.15.139:0/239531329 conn(0x7eff680f0ce0 msgr2=0x7eff680f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:47.179 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.189+0000 7eff6cf84700 1 -- 172.21.15.139:0/239531329 shutdown_connections 2021-11-20T06:03:47.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.189+0000 7eff6cf84700 1 -- 172.21.15.139:0/239531329 wait complete. 2021-11-20T06:03:47.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.190+0000 7eff6cf84700 1 Processor -- start 2021-11-20T06:03:47.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.190+0000 7eff6cf84700 1 -- start start 2021-11-20T06:03:47.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.190+0000 7eff6cf84700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff680f55a0 0x7eff6805ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:47.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.190+0000 7eff6cf84700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680f6840 0x7eff6810bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:47.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.190+0000 7eff6cf84700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6810c180 con 0x7eff680f6840 2021-11-20T06:03:47.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.190+0000 7eff6cf84700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6810c2c0 con 0x7eff680f55a0 2021-11-20T06:03:47.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.190+0000 7eff5ffff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680f6840 0x7eff6810bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:47.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.190+0000 7eff5ffff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680f6840 0x7eff6810bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57590/0 (socket says 172.21.15.139:57590) 2021-11-20T06:03:47.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.190+0000 7eff5ffff700 1 -- 172.21.15.139:0/481898786 learned_addr learned my addr 172.21.15.139:0/481898786 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:47.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.190+0000 7eff6659c700 1 --2- 172.21.15.139:0/481898786 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff680f55a0 0x7eff6805ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:47.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.190+0000 7eff5ffff700 1 -- 172.21.15.139:0/481898786 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff680f55a0 msgr2=0x7eff6805ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:47.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.190+0000 7eff5ffff700 1 --2- 172.21.15.139:0/481898786 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff680f55a0 0x7eff6805ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.183 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.190+0000 7eff5ffff700 1 -- 172.21.15.139:0/481898786 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff50005040 con 0x7eff680f6840 2021-11-20T06:03:47.183 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.191+0000 7eff6659c700 1 --2- 172.21.15.139:0/481898786 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff680f55a0 0x7eff6805ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:03:47.183 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.191+0000 7eff5ffff700 1 --2- 172.21.15.139:0/481898786 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680f6840 0x7eff6810bc90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7eff50009830 tx=0x7eff50009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:47.183 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.191+0000 7eff5effd700 1 -- 172.21.15.139:0/481898786 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff50018070 con 0x7eff680f6840 2021-11-20T06:03:47.184 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.191+0000 7eff6cf84700 1 -- 172.21.15.139:0/481898786 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6810c4c0 con 0x7eff680f6840 2021-11-20T06:03:47.184 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.191+0000 7eff5effd700 1 -- 172.21.15.139:0/481898786 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff50013410 con 0x7eff680f6840 2021-11-20T06:03:47.184 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.191+0000 7eff5effd700 1 -- 172.21.15.139:0/481898786 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff5001c3c0 con 0x7eff680f6840 2021-11-20T06:03:47.184 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.191+0000 7eff6cf84700 1 -- 172.21.15.139:0/481898786 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6810ca20 con 0x7eff680f6840 2021-11-20T06:03:47.184 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.192+0000 7eff6cf84700 1 -- 172.21.15.139:0/481898786 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff68027a00 con 0x7eff680f6840 2021-11-20T06:03:47.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.192+0000 7eff5effd700 1 -- 172.21.15.139:0/481898786 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7eff5001d420 con 0x7eff680f6840 2021-11-20T06:03:47.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.192+0000 7eff5effd700 1 --2- 172.21.15.139:0/481898786 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7eff5405c2a0 0x7eff5405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:47.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.193+0000 7eff5effd700 1 -- 172.21.15.139:0/481898786 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7eff50077ca0 con 0x7eff680f6840 2021-11-20T06:03:47.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.193+0000 7eff6659c700 1 --2- 172.21.15.139:0/481898786 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7eff5405c2a0 0x7eff5405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:47.186 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.193+0000 7eff6659c700 1 --2- 172.21.15.139:0/481898786 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7eff5405c2a0 0x7eff5405e760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7eff680f66a0 tx=0x7eff58009c00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:47.186 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.196+0000 7eff5effd700 1 -- 172.21.15.139:0/481898786 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff5007c070 con 0x7eff680f6840 2021-11-20T06:03:47.339 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.354+0000 7eff6cf84700 1 -- 172.21.15.139:0/481898786 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff680fdd30 con 0x7eff680f6840 2021-11-20T06:03:47.340 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.355+0000 7eff5effd700 1 -- 172.21.15.139:0/481898786 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+14123 (secure 0 0 0) 0x7eff5004a960 con 0x7eff680f6840 2021-11-20T06:03:47.340 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:03:47.340 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":41,"fsid":"d819f2cc-49c6-11ec-8c2c-001a4aab830c","created":"2021-11-20T05:58:33.540072+0000","modified":"2021-11-20T06:03:27.205472+0000","last_up_change":"2021-11-20T06:02:27.182023+0000","last_in_change":"2021-11-20T06:02:11.372444+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-20T06:01:48.405544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2021-11-20T06:03:23.869281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2021-11-20T06:03:24.435074+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fece5a6-7a49-451c-beb8-f90723d42033","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6803","nonce":22698595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6805","nonce":22698595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6809","nonce":22698595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":22698595},{"type":"v1","addr":"172.21.15.139:6807","nonce":22698595}]},"public_addr":"172.21.15.139:6803/22698595","cluster_addr":"172.21.15.139:6805/22698595","heartbeat_back_addr":"172.21.15.139:6809/22698595","heartbeat_front_addr":"172.21.15.139:6807/22698595","state":["exists","up"]},{"osd":1,"uuid":"ce18ef72-7c6c-43f7-8a0c-6904f7c9e606","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6811","nonce":2530906188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6813","nonce":2530906188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6817","nonce":2530906188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2530906188},{"type":"v1","addr":"172.21.15.139:6815","nonce":2530906188}]},"public_addr":"172.21.15.139:6811/2530906188","cluster_addr":"172.21.15.139:6813/2530906188","heartbeat_back_addr":"172.21.15.139:6817/2530906188","heartbeat_front_addr":"172.21.15.139:6815/2530906188","state":["exists","up"]},{"osd":2,"uuid":"80cfe585-e283-45a4-9dd7-f328ec00d21c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6819","nonce":1147042903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6821","nonce":1147042903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6825","nonce":1147042903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":1147042903},{"type":"v1","addr":"172.21.15.139:6823","nonce":1147042903}]},"public_addr":"172.21.15.139:6819/1147042903","cluster_addr":"172.21.15.139:6821/1147042903","heartbeat_back_addr":"172.21.15.139:6825/1147042903","heartbeat_front_addr":"172.21.15.139:6823/1147042903","state":["exists","up"]},{"osd":3,"uuid":"793e383f-4d5a-41cb-b5b3-b4e4e07d6b8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6801","nonce":927121970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6803","nonce":927121970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6807","nonce":927121970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":927121970},{"type":"v1","addr":"172.21.15.178:6805","nonce":927121970}]},"public_addr":"172.21.15.178:6801/927121970","cluster_addr":"172.21.15.178:6803/927121970","heartbeat_back_addr":"172.21.15.178:6807/927121970","heartbeat_front_addr":"172.21.15.178:6805/927121970","state":["exists","up"]},{"osd":4,"uuid":"3e453fbf-8ef9-4227-b3b8-cd6631bff6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6809","nonce":2601954583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6811","nonce":2601954583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6815","nonce":2601954583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":2601954583},{"type":"v1","addr":"172.21.15.178:6813","nonce":2601954583}]},"public_addr":"172.21.15.178:6809/2601954583","cluster_addr":"172.21.15.178:6811/2601954583","heartbeat_back_addr":"172.21.15.178:6815/2601954583","heartbeat_front_addr":"172.21.15.178:6813/2601954583","state":["exists","up"]},{"osd":5,"uuid":"b7d0eee4-3cb1-43dc-9c3f-f95593cf47cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6817","nonce":3932451665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6819","nonce":3932451665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6823","nonce":3932451665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3932451665},{"type":"v1","addr":"172.21.15.178:6821","nonce":3932451665}]},"public_addr":"172.21.15.178:6817/3932451665","cluster_addr":"172.21.15.178:6819/3932451665","heartbeat_back_addr":"172.21.15.178:6823/3932451665","heartbeat_front_addr":"172.21.15.178:6821/3932451665","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:01.290566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:19.499151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:37.169640+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:01:52.838242+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:02:09.060491+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-20T06:02:25.766679+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.139:0/3013495710":"2021-11-21T05:59:26.112534+0000","172.21.15.139:0/1461722096":"2021-11-21T05:59:13.525765+0000","172.21.15.139:0/1750602715":"2021-11-21T05:59:26.112534+0000","172.21.15.139:6800/3637001875":"2021-11-21T05:59:26.112534+0000","172.21.15.139:6801/1220659339":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6800/1220659339":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6801/3637001875":"2021-11-21T05:59:26.112534+0000","172.21.15.139:0/2668621147":"2021-11-21T05:59:13.525765+0000","172.21.15.139:6800/4049861874":"2021-11-21T05:58:51.653389+0000","172.21.15.139:0/2863654682":"2021-11-21T05:58:51.653389+0000","172.21.15.139:0/2447016509":"2021-11-21T05:58:51.653389+0000","172.21.15.139:6801/4049861874":"2021-11-21T05:58:51.653389+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-20T06:03:47.342 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.356+0000 7eff6cf84700 1 -- 172.21.15.139:0/481898786 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7eff5405c2a0 msgr2=0x7eff5405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:47.342 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.356+0000 7eff6cf84700 1 --2- 172.21.15.139:0/481898786 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7eff5405c2a0 0x7eff5405e760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7eff680f66a0 tx=0x7eff58009c00).stop 2021-11-20T06:03:47.342 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.356+0000 7eff6cf84700 1 -- 172.21.15.139:0/481898786 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680f6840 msgr2=0x7eff6810bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:47.342 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.356+0000 7eff6cf84700 1 --2- 172.21.15.139:0/481898786 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680f6840 0x7eff6810bc90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7eff50009830 tx=0x7eff50009b30).stop 2021-11-20T06:03:47.343 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.356+0000 7eff6cf84700 1 -- 172.21.15.139:0/481898786 shutdown_connections 2021-11-20T06:03:47.343 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.356+0000 7eff6cf84700 1 --2- 172.21.15.139:0/481898786 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7eff5405c2a0 0x7eff5405e760 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.343 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.356+0000 7eff6cf84700 1 --2- 172.21.15.139:0/481898786 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff680f55a0 0x7eff6805ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.343 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.356+0000 7eff6cf84700 1 --2- 172.21.15.139:0/481898786 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680f6840 0x7eff6810bc90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.344 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.356+0000 7eff6cf84700 1 -- 172.21.15.139:0/481898786 >> 172.21.15.139:0/481898786 conn(0x7eff680f0ce0 msgr2=0x7eff680fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:47.344 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.356+0000 7eff6cf84700 1 -- 172.21.15.139:0/481898786 shutdown_connections 2021-11-20T06:03:47.344 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.356+0000 7eff6cf84700 1 -- 172.21.15.139:0/481898786 wait complete. 2021-11-20T06:03:47.354 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-20T06:01:48.405544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2021-11-20T06:03:23.869281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2021-11-20T06:03:24.435074+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-11-20T06:03:47.355 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-11-20T06:03:47.477 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.486+0000 7f39fda40700 1 -- 172.21.15.139:0/339974452 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39f80f6680 msgr2=0x7f39f80f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:47.478 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.486+0000 7f39fda40700 1 --2- 172.21.15.139:0/339974452 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39f80f6680 0x7f39f80f6aa0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f39e8004660 tx=0x7f39e8009b30).stop 2021-11-20T06:03:47.478 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.486+0000 7f39fda40700 1 -- 172.21.15.139:0/339974452 shutdown_connections 2021-11-20T06:03:47.479 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.486+0000 7f39fda40700 1 --2- 172.21.15.139:0/339974452 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39f80f7920 0x7f39f80f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:03:47.480 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.486+0000 7f39fda40700 1 --2- 172.21.15.139:0/339974452 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39f80f6680 0x7f39f80f6aa0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.480 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.486+0000 7f39fda40700 1 -- 172.21.15.139:0/339974452 >> 172.21.15.139:0/339974452 conn(0x7f39f80f1de0 msgr2=0x7f39f80f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:47.480 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.486+0000 7f39fda40700 1 -- 172.21.15.139:0/339974452 shutdown_connections 2021-11-20T06:03:47.480 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.486+0000 7f39fda40700 1 -- 172.21.15.139:0/339974452 wait complete. 2021-11-20T06:03:47.481 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.487+0000 7f39fda40700 1 Processor -- start 2021-11-20T06:03:47.481 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.487+0000 7f39fda40700 1 -- start start 2021-11-20T06:03:47.481 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.487+0000 7f39fda40700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39f80f6680 0x7f39f8106e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:47.481 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.487+0000 7f39fda40700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39f80f7920 0x7f39f8107360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:47.482 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.487+0000 7f39fda40700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f81079a0 con 0x7f39f80f6680 2021-11-20T06:03:47.482 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.487+0000 7f39fda40700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f8107ae0 con 0x7f39f80f7920 2021-11-20T06:03:47.482 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.487+0000 7f39f6ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39f80f6680 0x7f39f8106e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:47.482 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.487+0000 7f39f6ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39f80f6680 0x7f39f8106e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57598/0 (socket says 172.21.15.139:57598) 2021-11-20T06:03:47.482 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.487+0000 7f39f6ffd700 1 -- 172.21.15.139:0/3309469787 learned_addr learned my addr 172.21.15.139:0/3309469787 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:47.483 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.488+0000 7f39f67fc700 1 --2- 172.21.15.139:0/3309469787 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39f80f7920 0x7f39f8107360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:47.483 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.488+0000 7f39f6ffd700 1 -- 172.21.15.139:0/3309469787 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39f80f7920 msgr2=0x7f39f8107360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:47.483 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.488+0000 7f39f6ffd700 1 --2- 172.21.15.139:0/3309469787 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39f80f7920 0x7f39f8107360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.484 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.488+0000 7f39f6ffd700 1 -- 172.21.15.139:0/3309469787 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e8005040 con 0x7f39f80f6680 2021-11-20T06:03:47.484 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.488+0000 7f39f6ffd700 1 --2- 172.21.15.139:0/3309469787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39f80f6680 0x7f39f8106e70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f39e8009f50 tx=0x7f39e8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:47.484 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.488+0000 7f39df7fe700 1 -- 172.21.15.139:0/3309469787 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39e8018070 con 0x7f39f80f6680 2021-11-20T06:03:47.484 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.488+0000 7f39df7fe700 1 -- 172.21.15.139:0/3309469787 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f39e8013410 con 0x7f39f80f6680 2021-11-20T06:03:47.485 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.488+0000 7f39df7fe700 1 -- 172.21.15.139:0/3309469787 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39e801c620 con 0x7f39f80f6680 2021-11-20T06:03:47.485 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.488+0000 7f39fda40700 1 -- 172.21.15.139:0/3309469787 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f810c520 con 0x7f39f80f6680 2021-11-20T06:03:47.485 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.488+0000 7f39fda40700 1 -- 172.21.15.139:0/3309469787 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f810ca80 con 0x7f39f80f6680 2021-11-20T06:03:47.486 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.489+0000 7f39fda40700 1 -- 172.21.15.139:0/3309469787 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f8027a00 con 0x7f39f80f6680 2021-11-20T06:03:47.486 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.489+0000 7f39df7fe700 1 -- 172.21.15.139:0/3309469787 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f39e801d420 con 0x7f39f80f6680 2021-11-20T06:03:47.486 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.489+0000 7f39df7fe700 1 --2- 172.21.15.139:0/3309469787 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f39e005c2a0 0x7f39e005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:47.486 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.490+0000 7f39df7fe700 1 -- 172.21.15.139:0/3309469787 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f39e8077dd0 con 0x7f39f80f6680 2021-11-20T06:03:47.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.490+0000 7f39f67fc700 1 --2- 172.21.15.139:0/3309469787 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f39e005c2a0 0x7f39e005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:47.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.490+0000 7f39f67fc700 1 --2- 172.21.15.139:0/3309469787 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f39e005c2a0 0x7f39e005e760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f39ec00b790 tx=0x7f39ec008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:47.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.493+0000 7f39df7fe700 1 -- 172.21.15.139:0/3309469787 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39e804aa30 con 0x7f39f80f6680 2021-11-20T06:03:47.637 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.652+0000 7f39fda40700 1 -- 172.21.15.139:0/3309469787 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f39f80fee90 con 0x7f39f80f6680 2021-11-20T06:03:47.638 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.652+0000 7f39df7fe700 1 -- 172.21.15.139:0/3309469787 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v41) v1 ==== 110+0+10 (secure 0 0 0) 0x7f39e804e050 con 0x7f39f80f6680 2021-11-20T06:03:47.638 INFO:teuthology.orchestra.run.smithi139.stdout:pg_num: 1 2021-11-20T06:03:47.639 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.654+0000 7f39fda40700 1 -- 172.21.15.139:0/3309469787 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f39e005c2a0 msgr2=0x7f39e005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:47.640 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.654+0000 7f39fda40700 1 --2- 172.21.15.139:0/3309469787 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f39e005c2a0 0x7f39e005e760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f39ec00b790 tx=0x7f39ec008040).stop 2021-11-20T06:03:47.640 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.654+0000 7f39fda40700 1 -- 172.21.15.139:0/3309469787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39f80f6680 msgr2=0x7f39f8106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:47.640 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.654+0000 7f39fda40700 1 --2- 172.21.15.139:0/3309469787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39f80f6680 0x7f39f8106e70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f39e8009f50 tx=0x7f39e8009b30).stop 2021-11-20T06:03:47.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.654+0000 7f39fda40700 1 -- 172.21.15.139:0/3309469787 shutdown_connections 2021-11-20T06:03:47.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.654+0000 7f39fda40700 1 --2- 172.21.15.139:0/3309469787 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f39e005c2a0 0x7f39e005e760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.654+0000 7f39fda40700 1 --2- 172.21.15.139:0/3309469787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f39f80f6680 0x7f39f8106e70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.654+0000 7f39fda40700 1 --2- 172.21.15.139:0/3309469787 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39f80f7920 0x7f39f8107360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.642 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.654+0000 7f39fda40700 1 -- 172.21.15.139:0/3309469787 >> 172.21.15.139:0/3309469787 conn(0x7f39f80f1de0 msgr2=0x7f39f80fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:47.642 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.654+0000 7f39fda40700 1 -- 172.21.15.139:0/3309469787 shutdown_connections 2021-11-20T06:03:47.642 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.654+0000 7f39fda40700 1 -- 172.21.15.139:0/3309469787 wait complete. 2021-11-20T06:03:47.650 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:47 smithi139 conmon[33501]: cluster 2021-11-20T06:03:46.212342+0000 mgr.smithi139.dplyxn 2021-11-20T06:03:47.650 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:47 smithi139 conmon[33501]: (mgr.14162) 199 : cluster [DBG] pgmap v160: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2021-11-20T06:03:47.650 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:47 smithi139 conmon[33501]: audit 2021-11-20T06:03:46.379555+0000 2021-11-20T06:03:47.651 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:47 smithi139 conmon[33501]: mon.smithi139 (mon.0) 536 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:47.651 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:47 smithi139 conmon[33501]: audit 2021-11-20T06:03:47.651 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:47 smithi139 conmon[33501]: 2021-11-20T06:03:46.383136+0000 mon.smithi139 (mon.0) 2021-11-20T06:03:47.651 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:47 smithi139 conmon[33501]: 537 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:47.652 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:47 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:03:47.652 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:47 smithi139 conmon[33501]: 20T06:03:47.355143+0000 mon.smithi139 (mon. 2021-11-20T06:03:47.652 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:47 smithi139 conmon[33501]: 0) 538 : audit [DBG] from='client.? 172.21.15.139:0/481898786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T06:03:47.653 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2021-11-20T06:03:47.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:47 smithi178 conmon[36600]: cluster 2021-11-20T06:03:46.212342+0000 mgr.smithi139.dplyxn 2021-11-20T06:03:47.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:47 smithi178 conmon[36600]: (mgr.14162) 199 : cluster [DBG] pgmap v160: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2021-11-20T06:03:47.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:47 smithi178 conmon[36600]: audit 2021-11-20T06:03:46.379555+0000 2021-11-20T06:03:47.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:47 smithi178 conmon[36600]: mon.smithi139 (mon.0) 536 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:47.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:47 smithi178 conmon[36600]: audit 2021-11-20T06:03:47.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:47 smithi178 conmon[36600]: 2021-11-20T06:03: 2021-11-20T06:03:47.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:47 smithi178 conmon[36600]: 46.383136+0000 mon.smithi139 (mon. 2021-11-20T06:03:47.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:47 smithi178 conmon[36600]: 0) 537 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:03:47.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:47 smithi178 conmon[36600]: 2021-11-20T06:03:47.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:47 smithi178 conmon[36600]: audit 2021-11- 2021-11-20T06:03:47.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:47 smithi178 conmon[36600]: 20T06:03:47.355143+0000 mon.smithi139 ( 2021-11-20T06:03:47.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:47 smithi178 conmon[36600]: mon.0) 538 : audit [DBG] from='client.? 172.21.15.139:0/481898786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-20T06:03:47.771 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.784+0000 7f306bd8f700 1 -- 172.21.15.139:0/1154358474 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30640f7920 msgr2=0x7f30640f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:47.772 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.784+0000 7f306bd8f700 1 --2- 172.21.15.139:0/1154358474 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30640f7920 0x7f30640f7d80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f3054004660 tx=0x7f3054009b30).stop 2021-11-20T06:03:47.772 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.784+0000 7f306bd8f700 1 -- 172.21.15.139:0/1154358474 shutdown_connections 2021-11-20T06:03:47.772 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.784+0000 7f306bd8f700 1 --2- 172.21.15.139:0/1154358474 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30640f7920 0x7f30640f7d80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.773 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.784+0000 7f306bd8f700 1 --2- 172.21.15.139:0/1154358474 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30640f6680 0x7f30640f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.773 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.784+0000 7f306bd8f700 1 -- 172.21.15.139:0/1154358474 >> 172.21.15.139:0/1154358474 conn(0x7f30640f1de0 msgr2=0x7f30640f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:47.773 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.784+0000 7f306bd8f700 1 -- 172.21.15.139:0/1154358474 shutdown_connections 2021-11-20T06:03:47.773 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.784+0000 7f306bd8f700 1 -- 172.21.15.139:0/1154358474 wait complete. 2021-11-20T06:03:47.773 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.784+0000 7f306bd8f700 1 Processor -- start 2021-11-20T06:03:47.774 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.784+0000 7f306bd8f700 1 -- start start 2021-11-20T06:03:47.774 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f306bd8f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30640f6680 0x7f306406b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:47.774 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f306bd8f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30640f7920 0x7f3064069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:47.774 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f306bd8f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306406bd70 con 0x7f30640f6680 2021-11-20T06:03:47.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f306bd8f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306406a310 con 0x7f30640f7920 2021-11-20T06:03:47.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f3069b2b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30640f6680 0x7f306406b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:47.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f3069b2b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30640f6680 0x7f306406b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57610/0 (socket says 172.21.15.139:57610) 2021-11-20T06:03:47.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f3069b2b700 1 -- 172.21.15.139:0/304309080 learned_addr learned my addr 172.21.15.139:0/304309080 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:47.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f306932a700 1 --2- 172.21.15.139:0/304309080 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30640f7920 0x7f3064069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:47.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f3069b2b700 1 -- 172.21.15.139:0/304309080 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30640f7920 msgr2=0x7f3064069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:47.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f3069b2b700 1 --2- 172.21.15.139:0/304309080 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30640f7920 0x7f3064069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f3069b2b700 1 -- 172.21.15.139:0/304309080 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3054005040 con 0x7f30640f6680 2021-11-20T06:03:47.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f3069b2b700 1 --2- 172.21.15.139:0/304309080 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30640f6680 0x7f306406b7c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f3060002700 tx=0x7f3060009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:47.777 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f305a7fc700 1 -- 172.21.15.139:0/304309080 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30600079c0 con 0x7f30640f6680 2021-11-20T06:03:47.777 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f305a7fc700 1 -- 172.21.15.139:0/304309080 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f306000f040 con 0x7f30640f6680 2021-11-20T06:03:47.777 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f306bd8f700 1 -- 172.21.15.139:0/304309080 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306406a590 con 0x7f30640f6680 2021-11-20T06:03:47.777 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f305a7fc700 1 -- 172.21.15.139:0/304309080 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3060014860 con 0x7f30640f6680 2021-11-20T06:03:47.778 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.785+0000 7f306bd8f700 1 -- 172.21.15.139:0/304309080 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306406ab50 con 0x7f30640f6680 2021-11-20T06:03:47.778 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.786+0000 7f306bd8f700 1 -- 172.21.15.139:0/304309080 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3064027a00 con 0x7f30640f6680 2021-11-20T06:03:47.778 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.787+0000 7f305a7fc700 1 -- 172.21.15.139:0/304309080 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f306001b060 con 0x7f30640f6680 2021-11-20T06:03:47.779 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.787+0000 7f305a7fc700 1 --2- 172.21.15.139:0/304309080 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f305005c2a0 0x7f305005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:47.780 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.787+0000 7f305a7fc700 1 -- 172.21.15.139:0/304309080 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f3060075d10 con 0x7f30640f6680 2021-11-20T06:03:47.780 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.787+0000 7f306932a700 1 --2- 172.21.15.139:0/304309080 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f305005c2a0 0x7f305005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:47.780 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.787+0000 7f306932a700 1 --2- 172.21.15.139:0/304309080 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f305005c2a0 0x7f305005e760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f3054009f50 tx=0x7f3054009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:47.780 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.791+0000 7f305a7fc700 1 -- 172.21.15.139:0/304309080 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30600489d0 con 0x7f30640f6680 2021-11-20T06:03:47.923 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.938+0000 7f306bd8f700 1 -- 172.21.15.139:0/304309080 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f30640fee10 con 0x7f30640f6680 2021-11-20T06:03:47.924 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.938+0000 7f305a7fc700 1 -- 172.21.15.139:0/304309080 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7f306004bff0 con 0x7f30640f6680 2021-11-20T06:03:47.924 INFO:teuthology.orchestra.run.smithi139.stdout:pg_num: 32 2021-11-20T06:03:47.925 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.940+0000 7f306bd8f700 1 -- 172.21.15.139:0/304309080 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f305005c2a0 msgr2=0x7f305005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:47.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.940+0000 7f306bd8f700 1 --2- 172.21.15.139:0/304309080 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f305005c2a0 0x7f305005e760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f3054009f50 tx=0x7f3054009b30).stop 2021-11-20T06:03:47.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.940+0000 7f306bd8f700 1 -- 172.21.15.139:0/304309080 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30640f6680 msgr2=0x7f306406b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:47.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.940+0000 7f306bd8f700 1 --2- 172.21.15.139:0/304309080 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30640f6680 0x7f306406b7c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f3060002700 tx=0x7f3060009d80).stop 2021-11-20T06:03:47.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.940+0000 7f306bd8f700 1 -- 172.21.15.139:0/304309080 shutdown_connections 2021-11-20T06:03:47.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.940+0000 7f306bd8f700 1 --2- 172.21.15.139:0/304309080 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f305005c2a0 0x7f305005e760 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.940+0000 7f306bd8f700 1 --2- 172.21.15.139:0/304309080 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30640f6680 0x7f306406b7c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.940+0000 7f306bd8f700 1 --2- 172.21.15.139:0/304309080 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30640f7920 0x7f3064069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:47.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.940+0000 7f306bd8f700 1 -- 172.21.15.139:0/304309080 >> 172.21.15.139:0/304309080 conn(0x7f30640f1de0 msgr2=0x7f30640fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:47.928 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.940+0000 7f306bd8f700 1 -- 172.21.15.139:0/304309080 shutdown_connections 2021-11-20T06:03:47.928 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:47.940+0000 7f306bd8f700 1 -- 172.21.15.139:0/304309080 wait complete. 2021-11-20T06:03:47.937 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2021-11-20T06:03:48.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.077+0000 7fb7f676b700 1 -- 172.21.15.139:0/3676233648 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7f00f7b40 msgr2=0x7fb7f00f7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:48.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.077+0000 7fb7f676b700 1 --2- 172.21.15.139:0/3676233648 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7f00f7b40 0x7fb7f00f7fa0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fb7e4004660 tx=0x7fb7e4009b30).stop 2021-11-20T06:03:48.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.077+0000 7fb7f676b700 1 -- 172.21.15.139:0/3676233648 shutdown_connections 2021-11-20T06:03:48.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.077+0000 7fb7f676b700 1 --2- 172.21.15.139:0/3676233648 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7f00f7b40 0x7fb7f00f7fa0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:48.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.077+0000 7fb7f676b700 1 --2- 172.21.15.139:0/3676233648 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7f00f68a0 0x7fb7f00f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:48.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.077+0000 7fb7f676b700 1 -- 172.21.15.139:0/3676233648 >> 172.21.15.139:0/3676233648 conn(0x7fb7f00f1fc0 msgr2=0x7fb7f00f4420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:48.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.077+0000 7fb7f676b700 1 -- 172.21.15.139:0/3676233648 shutdown_connections 2021-11-20T06:03:48.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.077+0000 7fb7f676b700 1 -- 172.21.15.139:0/3676233648 wait complete. 2021-11-20T06:03:48.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.078+0000 7fb7f676b700 1 Processor -- start 2021-11-20T06:03:48.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.078+0000 7fb7f676b700 1 -- start start 2021-11-20T06:03:48.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.078+0000 7fb7f676b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7f00f68a0 0x7fb7f006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:48.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.078+0000 7fb7f676b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7f00f7b40 0x7fb7f0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:48.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.078+0000 7fb7f676b700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f006bd70 con 0x7fb7f00f68a0 2021-11-20T06:03:48.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.078+0000 7fb7f676b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f006a310 con 0x7fb7f00f7b40 2021-11-20T06:03:48.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.078+0000 7fb7effff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7f00f68a0 0x7fb7f006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:48.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.078+0000 7fb7effff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7f00f68a0 0x7fb7f006b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57622/0 (socket says 172.21.15.139:57622) 2021-11-20T06:03:48.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.078+0000 7fb7effff700 1 -- 172.21.15.139:0/4233929828 learned_addr learned my addr 172.21.15.139:0/4233929828 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:48.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.078+0000 7fb7effff700 1 -- 172.21.15.139:0/4233929828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7f00f7b40 msgr2=0x7fb7f0069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:48.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.078+0000 7fb7effff700 1 --2- 172.21.15.139:0/4233929828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7f00f7b40 0x7fb7f0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:03:48.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.078+0000 7fb7effff700 1 -- 172.21.15.139:0/4233929828 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e4005040 con 0x7fb7f00f68a0 2021-11-20T06:03:48.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.079+0000 7fb7effff700 1 --2- 172.21.15.139:0/4233929828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7f00f68a0 0x7fb7f006b7c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fb7e0002700 tx=0x7fb7e0009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:48.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.079+0000 7fb7ecff9700 1 -- 172.21.15.139:0/4233929828 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7e0009320 con 0x7fb7f00f68a0 2021-11-20T06:03:48.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.079+0000 7fb7ecff9700 1 -- 172.21.15.139:0/4233929828 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7e000f040 con 0x7fb7f00f68a0 2021-11-20T06:03:48.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.079+0000 7fb7f676b700 1 -- 172.21.15.139:0/4233929828 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f006a590 con 0x7fb7f00f68a0 2021-11-20T06:03:48.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.079+0000 7fb7ecff9700 1 -- 172.21.15.139:0/4233929828 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7e00148e0 con 0x7fb7f00f68a0 2021-11-20T06:03:48.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.079+0000 7fb7f676b700 1 -- 172.21.15.139:0/4233929828 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f006ab50 con 0x7fb7f00f68a0 2021-11-20T06:03:48.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.080+0000 7fb7f676b700 1 -- 172.21.15.139:0/4233929828 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f005ea10 con 0x7fb7f00f68a0 2021-11-20T06:03:48.072 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.080+0000 7fb7ecff9700 1 -- 172.21.15.139:0/4233929828 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb7e0009480 con 0x7fb7f00f68a0 2021-11-20T06:03:48.073 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.081+0000 7fb7ecff9700 1 --2- 172.21.15.139:0/4233929828 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb7d805c310 0x7fb7d805e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:48.073 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.081+0000 7fb7ecff9700 1 -- 172.21.15.139:0/4233929828 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fb7e0076640 con 0x7fb7f00f68a0 2021-11-20T06:03:48.073 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.081+0000 7fb7ef7fe700 1 --2- 172.21.15.139:0/4233929828 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb7d805c310 0x7fb7d805e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:48.073 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.081+0000 7fb7ef7fe700 1 --2- 172.21.15.139:0/4233929828 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb7d805c310 0x7fb7d805e7d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fb7e4000c00 tx=0x7fb7e4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:48.074 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.084+0000 7fb7ecff9700 1 -- 172.21.15.139:0/4233929828 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7e00492c0 con 0x7fb7f00f68a0 2021-11-20T06:03:48.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.232+0000 7fb7f676b700 1 -- 172.21.15.139:0/4233929828 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fb7f0027a00 con 0x7fb7f00f68a0 2021-11-20T06:03:48.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.233+0000 7fb7ecff9700 1 -- 172.21.15.139:0/4233929828 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7fb7e004c8e0 con 0x7fb7f00f68a0 2021-11-20T06:03:48.220 INFO:teuthology.orchestra.run.smithi139.stdout:pg_num: 32 2021-11-20T06:03:48.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.235+0000 7fb7f676b700 1 -- 172.21.15.139:0/4233929828 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb7d805c310 msgr2=0x7fb7d805e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:48.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.235+0000 7fb7f676b700 1 --2- 172.21.15.139:0/4233929828 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb7d805c310 0x7fb7d805e7d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fb7e4000c00 tx=0x7fb7e4009b30).stop 2021-11-20T06:03:48.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.235+0000 7fb7f676b700 1 -- 172.21.15.139:0/4233929828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7f00f68a0 msgr2=0x7fb7f006b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:48.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.235+0000 7fb7f676b700 1 --2- 172.21.15.139:0/4233929828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7f00f68a0 0x7fb7f006b7c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fb7e0002700 tx=0x7fb7e0009e00).stop 2021-11-20T06:03:48.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.235+0000 7fb7f676b700 1 -- 172.21.15.139:0/4233929828 shutdown_connections 2021-11-20T06:03:48.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.235+0000 7fb7f676b700 1 --2- 172.21.15.139:0/4233929828 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb7d805c310 0x7fb7d805e7d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:48.223 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.235+0000 7fb7f676b700 1 --2- 172.21.15.139:0/4233929828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb7f00f68a0 0x7fb7f006b7c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:48.223 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.235+0000 7fb7f676b700 1 --2- 172.21.15.139:0/4233929828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7f00f7b40 0x7fb7f0069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:03:48.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.235+0000 7fb7f676b700 1 -- 172.21.15.139:0/4233929828 >> 172.21.15.139:0/4233929828 conn(0x7fb7f00f1fc0 msgr2=0x7fb7f00fb590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:48.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.235+0000 7fb7f676b700 1 -- 172.21.15.139:0/4233929828 shutdown_connections 2021-11-20T06:03:48.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.235+0000 7fb7f676b700 1 -- 172.21.15.139:0/4233929828 wait complete. 2021-11-20T06:03:48.232 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-11-20T06:03:48.232 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:03:48.349 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.361+0000 7f5686a26700 1 -- 172.21.15.139:0/1999158941 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56800f6830 msgr2=0x7f56800f6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:48.349 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.361+0000 7f5686a26700 1 --2- 172.21.15.139:0/1999158941 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56800f6830 0x7f56800f6c90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f5674004660 tx=0x7f5674009b30).stop 2021-11-20T06:03:48.349 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.361+0000 7f5686a26700 1 -- 172.21.15.139:0/1999158941 shutdown_connections 2021-11-20T06:03:48.350 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.361+0000 7f5686a26700 1 --2- 172.21.15.139:0/1999158941 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56800f6830 0x7f56800f6c90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:48.350 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.361+0000 7f5686a26700 1 --2- 172.21.15.139:0/1999158941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56800f5590 0x7f56800f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:48.350 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.361+0000 7f5686a26700 1 -- 172.21.15.139:0/1999158941 >> 172.21.15.139:0/1999158941 conn(0x7f56800f0cf0 msgr2=0x7f56800f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:48.350 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.362+0000 7f5686a26700 1 -- 172.21.15.139:0/1999158941 shutdown_connections 2021-11-20T06:03:48.350 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.362+0000 7f5686a26700 1 -- 172.21.15.139:0/1999158941 wait complete. 2021-11-20T06:03:48.351 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.362+0000 7f5686a26700 1 Processor -- start 2021-11-20T06:03:48.351 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.362+0000 7f5686a26700 1 -- start start 2021-11-20T06:03:48.351 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.362+0000 7f5686a26700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56800f5590 0x7f5680105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:48.351 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.362+0000 7f5686a26700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56800f6830 0x7f5680106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:48.351 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.362+0000 7f5686a26700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5680106890 con 0x7f56800f6830 2021-11-20T06:03:48.352 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.362+0000 7f5686a26700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56801069d0 con 0x7f56800f5590 2021-11-20T06:03:48.352 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.362+0000 7f567f7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56800f6830 0x7f5680106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:48.352 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.362+0000 7f567f7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56800f6830 0x7f5680106290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57634/0 (socket says 172.21.15.139:57634) 2021-11-20T06:03:48.352 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.362+0000 7f567f7fe700 1 -- 172.21.15.139:0/3053137380 learned_addr learned my addr 172.21.15.139:0/3053137380 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:48.353 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.362+0000 7f567ffff700 1 --2- 172.21.15.139:0/3053137380 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56800f5590 0x7f5680105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:48.353 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.363+0000 7f567f7fe700 1 -- 172.21.15.139:0/3053137380 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56800f5590 msgr2=0x7f5680105da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:48.353 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.363+0000 7f567f7fe700 1 --2- 172.21.15.139:0/3053137380 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56800f5590 0x7f5680105da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:48.354 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.363+0000 7f567f7fe700 1 -- 172.21.15.139:0/3053137380 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5674005040 con 0x7f56800f6830 2021-11-20T06:03:48.354 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.363+0000 7f567f7fe700 1 --2- 172.21.15.139:0/3053137380 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56800f6830 0x7f5680106290 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f5674009830 tx=0x7f5674009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:48.354 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.363+0000 7f567cff9700 1 -- 172.21.15.139:0/3053137380 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f567401a070 con 0x7f56800f6830 2021-11-20T06:03:48.354 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.363+0000 7f567cff9700 1 -- 172.21.15.139:0/3053137380 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5674013410 con 0x7f56800f6830 2021-11-20T06:03:48.354 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.363+0000 7f567cff9700 1 -- 172.21.15.139:0/3053137380 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f567401f880 con 0x7f56800f6830 2021-11-20T06:03:48.355 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.363+0000 7f5686a26700 1 -- 172.21.15.139:0/3053137380 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568005fa80 con 0x7f56800f6830 2021-11-20T06:03:48.355 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.363+0000 7f5686a26700 1 -- 172.21.15.139:0/3053137380 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568005ffe0 con 0x7f56800f6830 2021-11-20T06:03:48.356 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.364+0000 7f5686a26700 1 -- 172.21.15.139:0/3053137380 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5680046330 con 0x7f56800f6830 2021-11-20T06:03:48.356 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.364+0000 7f567cff9700 1 -- 172.21.15.139:0/3053137380 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5674012c10 con 0x7f56800f6830 2021-11-20T06:03:48.356 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.365+0000 7f567cff9700 1 --2- 172.21.15.139:0/3053137380 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f566805c2a0 0x7f566805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:48.357 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.365+0000 7f567cff9700 1 -- 172.21.15.139:0/3053137380 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f567407a020 con 0x7f56800f6830 2021-11-20T06:03:48.357 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.365+0000 7f567ffff700 1 --2- 172.21.15.139:0/3053137380 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f566805c2a0 0x7f566805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:48.357 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.365+0000 7f567ffff700 1 --2- 172.21.15.139:0/3053137380 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f566805c2a0 0x7f566805e760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5680107290 tx=0x7f567000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:48.357 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.369+0000 7f567cff9700 1 -- 172.21.15.139:0/3053137380 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f567404cce0 con 0x7f56800f6830 2021-11-20T06:03:48.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:48 smithi139 conmon[33501]: audit 2021-11-20T06:03:47.653140+0000 mon.smithi139 2021-11-20T06:03:48.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:48 smithi139 conmon[33501]: (mon.0) 539 : audit [DBG] from='client.? 172.21.15.139:0/3309469787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T06:03:48.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:48 smithi139 conmon[33501]: audit 2021-11-20T06:03:47.939199+0000 mon.smithi139 (mon.0) 540 : audit [DBG] from='client.? 172.21.15.139:0/304309080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T06:03:48.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:48 smithi139 conmon[33501]: audit 2021-11-20T06:03:48.233821+0000 mon.smithi139 (mon.0) 541 : audit 2021-11-20T06:03:48.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:48 smithi139 conmon[33501]: [DBG] from='client.? 172.21.15.139:0/4233929828' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T06:03:48.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.553+0000 7f5686a26700 1 -- 172.21.15.139:0/3053137380 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56800fdd20 con 0x7f56800f6830 2021-11-20T06:03:48.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.554+0000 7f567cff9700 1 -- 172.21.15.139:0/3053137380 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2945 (secure 0 0 0) 0x7f5674050300 con 0x7f56800f6830 2021-11-20T06:03:48.539 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:03:48.540 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14442,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6829/2967963544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2967963544},{"type":"v1","addr":"172.21.15.139:6829","nonce":2967963544}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":14},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13}],"filesystems":[{"mdsmap":{"epoch":11,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:03:34.486673+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24237},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14426":{"gid":14426,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.139:6827/724620583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":724620583},{"type":"v1","addr":"172.21.15.139:6827","nonce":724620583}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24237":{"gid":24237,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.178:6825/1867304351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1867304351},{"type":"v1","addr":"172.21.15.178:6825","nonce":1867304351}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:03:48.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.555+0000 7f5686a26700 1 -- 172.21.15.139:0/3053137380 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f566805c2a0 msgr2=0x7f566805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:48.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.555+0000 7f5686a26700 1 --2- 172.21.15.139:0/3053137380 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f566805c2a0 0x7f566805e760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5680107290 tx=0x7f567000b040).stop 2021-11-20T06:03:48.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.555+0000 7f5686a26700 1 -- 172.21.15.139:0/3053137380 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56800f6830 msgr2=0x7f5680106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:48.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.555+0000 7f5686a26700 1 --2- 172.21.15.139:0/3053137380 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56800f6830 0x7f5680106290 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f5674009830 tx=0x7f5674009b30).stop 2021-11-20T06:03:48.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.555+0000 7f5686a26700 1 -- 172.21.15.139:0/3053137380 shutdown_connections 2021-11-20T06:03:48.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.555+0000 7f5686a26700 1 --2- 172.21.15.139:0/3053137380 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f566805c2a0 0x7f566805e760 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:48.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.555+0000 7f5686a26700 1 --2- 172.21.15.139:0/3053137380 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56800f5590 0x7f5680105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:48.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.555+0000 7f5686a26700 1 --2- 172.21.15.139:0/3053137380 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f56800f6830 0x7f5680106290 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:48.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.555+0000 7f5686a26700 1 -- 172.21.15.139:0/3053137380 >> 172.21.15.139:0/3053137380 conn(0x7f56800f0cf0 msgr2=0x7f56800fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:48.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.556+0000 7f5686a26700 1 -- 172.21.15.139:0/3053137380 shutdown_connections 2021-11-20T06:03:48.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.556+0000 7f5686a26700 1 -- 172.21.15.139:0/3053137380 wait complete. 2021-11-20T06:03:48.544 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 14 2021-11-20T06:03:48.553 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 11, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:03:34.486673+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24237}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14426': {'gid': 14426, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 2, 'addr': '172.21.15.139:6827/724620583', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 724620583}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 724620583}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24237': {'gid': 24237, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.178:6825/1867304351', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 1867304351}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 1867304351}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:03:48.553 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-11-20T06:03:48.554 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.cephfs.smithi178.blvkbx status 2021-11-20T06:03:48.669 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.682+0000 7f259d35b700 1 -- 172.21.15.139:0/3831118389 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25980f6680 msgr2=0x7f25980f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:48.669 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.682+0000 7f259d35b700 1 --2- 172.21.15.139:0/3831118389 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25980f6680 0x7f25980f6aa0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f2588004660 tx=0x7f2588009b30).stop 2021-11-20T06:03:48.670 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.682+0000 7f259d35b700 1 -- 172.21.15.139:0/3831118389 shutdown_connections 2021-11-20T06:03:48.670 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.682+0000 7f259d35b700 1 --2- 172.21.15.139:0/3831118389 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25980f7920 0x7f25980f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:48.670 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.682+0000 7f259d35b700 1 --2- 172.21.15.139:0/3831118389 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25980f6680 0x7f25980f6aa0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:48.670 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.682+0000 7f259d35b700 1 -- 172.21.15.139:0/3831118389 >> 172.21.15.139:0/3831118389 conn(0x7f25980f1de0 msgr2=0x7f25980f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:48.670 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.682+0000 7f259d35b700 1 -- 172.21.15.139:0/3831118389 shutdown_connections 2021-11-20T06:03:48.671 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.682+0000 7f259d35b700 1 -- 172.21.15.139:0/3831118389 wait complete. 2021-11-20T06:03:48.671 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f259d35b700 1 Processor -- start 2021-11-20T06:03:48.671 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f259d35b700 1 -- start start 2021-11-20T06:03:48.671 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f259d35b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25980f6680 0x7f2598106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:48.672 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f259d35b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25980f7920 0x7f25981073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:48.672 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f259d35b700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2598107960 con 0x7f25980f6680 2021-11-20T06:03:48.672 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f259d35b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2598107aa0 con 0x7f25980f7920 2021-11-20T06:03:48.672 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f2596ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25980f6680 0x7f2598106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:48.673 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f2596ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25980f6680 0x7f2598106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57642/0 (socket says 172.21.15.139:57642) 2021-11-20T06:03:48.673 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f2596ffd700 1 -- 172.21.15.139:0/1332294078 learned_addr learned my addr 172.21.15.139:0/1332294078 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:48.673 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f25967fc700 1 --2- 172.21.15.139:0/1332294078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25980f7920 0x7f25981073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:48.673 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f2596ffd700 1 -- 172.21.15.139:0/1332294078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25980f7920 msgr2=0x7f25981073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:48.674 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f2596ffd700 1 --2- 172.21.15.139:0/1332294078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25980f7920 0x7f25981073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:48.674 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f2596ffd700 1 -- 172.21.15.139:0/1332294078 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2588005040 con 0x7f25980f6680 2021-11-20T06:03:48.674 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f2596ffd700 1 --2- 172.21.15.139:0/1332294078 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25980f6680 0x7f2598106ec0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f2588004e30 tx=0x7f2588009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:48.674 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f257f7fe700 1 -- 172.21.15.139:0/1332294078 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2588018070 con 0x7f25980f6680 2021-11-20T06:03:48.674 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f259d35b700 1 -- 172.21.15.139:0/1332294078 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259810c4e0 con 0x7f25980f6680 2021-11-20T06:03:48.675 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f257f7fe700 1 -- 172.21.15.139:0/1332294078 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2588013410 con 0x7f25980f6680 2021-11-20T06:03:48.675 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.683+0000 7f257f7fe700 1 -- 172.21.15.139:0/1332294078 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f258801c3c0 con 0x7f25980f6680 2021-11-20T06:03:48.675 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.684+0000 7f259d35b700 1 -- 172.21.15.139:0/1332294078 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259810caa0 con 0x7f25980f6680 2021-11-20T06:03:48.682 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.685+0000 7f257f7fe700 1 -- 172.21.15.139:0/1332294078 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f258801d420 con 0x7f25980f6680 2021-11-20T06:03:48.682 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.685+0000 7f257f7fe700 1 --2- 172.21.15.139:0/1332294078 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f258005c2a0 0x7f258005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:48.683 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.685+0000 7f257f7fe700 1 -- 172.21.15.139:0/1332294078 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f2588077b60 con 0x7f25980f6680 2021-11-20T06:03:48.683 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.685+0000 7f25967fc700 1 --2- 172.21.15.139:0/1332294078 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f258005c2a0 0x7f258005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:48.683 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.686+0000 7f25967fc700 1 --2- 172.21.15.139:0/1332294078 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f258005c2a0 0x7f258005e760 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f258c0096c0 tx=0x7f258c009c20).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:48.683 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.687+0000 7f25aef06b80 1 Processor -- start 2021-11-20T06:03:48.684 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.687+0000 7f25aef06b80 1 -- start start 2021-11-20T06:03:48.684 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.687+0000 7f25aef06b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cd16c111d0 0x55cd16da3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:48.684 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.687+0000 7f25aef06b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55cd16da3fe0 0x55cd16da4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:48.684 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.687+0000 7f25aef06b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd16ce24d0 con 0x55cd16c111d0 2021-11-20T06:03:48.684 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.687+0000 7f25aef06b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd16cf63e0 con 0x55cd16da3fe0 2021-11-20T06:03:48.685 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.687+0000 7f2596ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cd16c111d0 0x55cd16da3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:48.685 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.687+0000 7f2596ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cd16c111d0 0x55cd16da3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57648/0 (socket says 172.21.15.139:57648) 2021-11-20T06:03:48.685 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.687+0000 7f2596ffd700 1 -- 172.21.15.139:0/3155249188 learned_addr learned my addr 172.21.15.139:0/3155249188 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:48.685 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.687+0000 7f2596ffd700 1 -- 172.21.15.139:0/3155249188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55cd16da3fe0 msgr2=0x55cd16da4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T06:03:48.686 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.687+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3155249188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55cd16da3fe0 0x55cd16da4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:03:48.686 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.688+0000 7f2596ffd700 1 -- 172.21.15.139:0/3155249188 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd16d723c0 con 0x55cd16c111d0 2021-11-20T06:03:48.686 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.688+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3155249188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cd16c111d0 0x55cd16da3af0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f2588013690 tx=0x7f2588013620).ready entity=mon.0 client_cookie=91a94231b7ced7c3 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:48.686 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.688+0000 7f257cff9700 1 -- 172.21.15.139:0/3155249188 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f258804e480 con 0x55cd16c111d0 2021-11-20T06:03:48.686 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.688+0000 7f257cff9700 1 -- 172.21.15.139:0/3155249188 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f258804c620 con 0x55cd16c111d0 2021-11-20T06:03:48.687 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.688+0000 7f257cff9700 1 -- 172.21.15.139:0/3155249188 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f258801b5b0 con 0x55cd16c111d0 2021-11-20T06:03:48.687 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.688+0000 7f25aef06b80 1 -- 172.21.15.139:0/3155249188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cd16c111d0 msgr2=0x55cd16da3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:03:48.687 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.688+0000 7f25aef06b80 1 --2- 172.21.15.139:0/3155249188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cd16c111d0 0x55cd16da3af0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f2588013690 tx=0x7f2588013620).stop 2021-11-20T06:03:48.687 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.688+0000 7f25aef06b80 1 -- 172.21.15.139:0/3155249188 shutdown_connections 2021-11-20T06:03:48.687 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.688+0000 7f25aef06b80 1 --2- 172.21.15.139:0/3155249188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55cd16da3fe0 0x55cd16da4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:03:48.688 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.688+0000 7f25aef06b80 1 --2- 172.21.15.139:0/3155249188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cd16c111d0 0x55cd16da3af0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:48.688 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.688+0000 7f25aef06b80 1 -- 172.21.15.139:0/3155249188 >> 172.21.15.139:0/3155249188 conn(0x55cd16d699b0 msgr2=0x55cd16d9efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:03:48.688 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.688+0000 7f25aef06b80 1 -- 172.21.15.139:0/3155249188 shutdown_connections 2021-11-20T06:03:48.688 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f25aef06b80 1 -- 172.21.15.139:0/3155249188 wait complete. 2021-11-20T06:03:48.689 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f25aef06b80 1 -- start start 2021-11-20T06:03:48.689 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f25aef06b80 1 Processor -- start 2021-11-20T06:03:48.689 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f25aef06b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55cd16da3fe0 0x55cd16dacb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:48.689 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f25aef06b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cd16dad020 0x55cd16db20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:48.689 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f25aef06b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd16cf6520 con 0x55cd16dad020 2021-11-20T06:03:48.690 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f25aef06b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd16cf6aa0 con 0x55cd16da3fe0 2021-11-20T06:03:48.690 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f25aef06b80 10 client.0 ll_register_callbacks cb 0x55cd16d63700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-20T06:03:48.690 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f25967fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cd16dad020 0x55cd16db20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:48.690 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f25967fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cd16dad020 0x55cd16db20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57654/0 (socket says 172.21.15.139:57654) 2021-11-20T06:03:48.691 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f25967fc700 1 -- 172.21.15.139:0/3754453719 learned_addr learned my addr 172.21.15.139:0/3754453719 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:03:48.691 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55cd16da3fe0 0x55cd16dacb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:03:48.691 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f25967fc700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55cd16da3fe0 msgr2=0x55cd16dacb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-20T06:03:48.691 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f25967fc700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55cd16da3fe0 0x55cd16dacb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:03:48.691 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f25967fc700 1 -- 172.21.15.139:0/3754453719 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259810caa0 con 0x55cd16dad020 2021-11-20T06:03:48.692 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55cd16da3fe0 0x55cd16dacb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:03:48.692 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.689+0000 7f25967fc700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cd16dad020 0x55cd16db20c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f258c0196b0 tx=0x7f258c00b000).ready entity=mon.0 client_cookie=d423f45bb87320b0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:03:48.692 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f2575ffb700 10 client.0 ms_handle_connect on v2:172.21.15.139:3300/0 2021-11-20T06:03:48.692 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f2575ffb700 1 -- 172.21.15.139:0/3754453719 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f258c017850 con 0x55cd16dad020 2021-11-20T06:03:48.693 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f2575ffb700 1 -- 172.21.15.139:0/3754453719 <== mon.0 v2:172.21.15.139:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f258c0179b0 con 0x55cd16dad020 2021-11-20T06:03:48.693 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f25aef06b80 1 -- 172.21.15.139:0/3754453719 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cd16d71ce0 con 0x55cd16dad020 2021-11-20T06:03:48.693 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f2575ffb700 1 -- 172.21.15.139:0/3754453719 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7f258c017bc0 con 0x55cd16dad020 2021-11-20T06:03:48.693 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f25aef06b80 10 client.14534 fetch_fsmap learned FSMap version 14 2021-11-20T06:03:48.693 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f25aef06b80 1 -- 172.21.15.139:0/3754453719 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=3+}) v3 -- 0x55cd16ce5630 con 0x55cd16dad020 2021-11-20T06:03:48.694 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f2575ffb700 1 -- 172.21.15.139:0/3754453719 <== mon.0 v2:172.21.15.139:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x55cd16d71ce0 con 0x55cd16dad020 2021-11-20T06:03:48.694 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f2575ffb700 1 -- 172.21.15.139:0/3754453719 <== mon.0 v2:172.21.15.139:3300/0 5 ==== fsmap(e 14) v1 ==== 1712+0+0 (secure 0 0 0) 0x7f258c020070 con 0x55cd16dad020 2021-11-20T06:03:48.694 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f25aef06b80 10 client.14534 fetch_fsmap finished waiting for FSMap version 14 2021-11-20T06:03:48.694 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f25aef06b80 10 client.14534 resolve_mds: resolved name 'cephfs.smithi178.blvkbx' to daemon mds.cephfs.smithi178.blvkbx 2021-11-20T06:03:48.695 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f25aef06b80 20 client.14534 populate_metadata read hostname 'smithi139' 2021-11-20T06:03:48.695 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f25aef06b80 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-20T06:03:48.695 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f25aef06b80 4 client.14534 mds_command: new command op to 24237 tid=0[{"prefix": "get_command_descriptions"}] 2021-11-20T06:03:48.695 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f25aef06b80 1 -- 172.21.15.139:0/3754453719 --> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd16cecdf0 con 0x55cd16db5d80 2021-11-20T06:03:48.695 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:48.696 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f2575ffb700 1 client.14534 ms_handle_refused on v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:48.696 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.690+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-20T06:03:48.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:48 smithi178 conmon[36600]: audit 2021-11-20T06:03:47.653140+0000 mon.smithi139 (mon.0) 539 : 2021-11-20T06:03:48.697 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:48 smithi178 conmon[36600]: audit [DBG] from='client.? 172.21.15.139:0/3309469787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-20T06:03:48.697 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:48 smithi178 conmon[36600]: audit 2021-11-20T06:03:47.939199+0000 mon.smithi139 (mon.0) 540 : audit [DBG] from='client.? 172.21.15.139:0/304309080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-20T06:03:48.697 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:48 smithi178 conmon[36600]: audit 2021-11-20T06:03:48.233821+0000 mon.smithi139 (mon.0) 541 : audit [DBG] from='client.? 172.21.15.139:0/4233929828' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-20T06:03:48.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.891+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:48.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.891+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-20T06:03:48.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:48.891+0000 7f2575ffb700 1 client.14534 ms_handle_refused on v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:49.277 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:49.292+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:49.277 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:49.292+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-20T06:03:49.277 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:49.292+0000 7f2575ffb700 1 client.14534 ms_handle_refused on v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:49.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:49 smithi139 conmon[33501]: cluster 2021-11-20T06:03:48.212821+0000 mgr.smithi139.dplyxn (mgr.14162) 200 : cluster 2021-11-20T06:03:49.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:49 smithi139 conmon[33501]: [DBG] pgmap v161: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2021-11-20T06:03:49.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:49 smithi139 conmon[33501]: audit 2021-11-20T06:03:48.554479+0000 mon.smithi139 (mon.0) 542 : audit [DBG] from='client.? 172.21.15.139:0/3053137380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:03:49.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:49 smithi178 conmon[36600]: cluster 2021-11-20T06:03:48.212821+0000 mgr.smithi139.dplyxn (mgr.14162) 200 : cluster [DBG] 2021-11-20T06:03:49.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:49 smithi178 conmon[36600]: pgmap v161: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2021-11-20T06:03:49.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:49 smithi178 conmon[36600]: audit 2021-11-20T06:03:48.554479+0000 mon.smithi139 (mon.0) 542 : audit [DBG] from='client.? 172.21.15.139:0/3053137380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:03:50.078 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:50.093+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:50.079 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:50.093+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-20T06:03:50.079 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:50.093+0000 7f2575ffb700 1 client.14534 ms_handle_refused on v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:51.679 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:51.694+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:51.679 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:51.694+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-20T06:03:51.679 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:51.694+0000 7f2575ffb700 1 client.14534 ms_handle_refused on v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:51.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:51 smithi178 conmon[36600]: cluster 2021-11-20T06:03:50.213413+0000 mgr.smithi139.dplyxn (mgr.14162) 201 : cluster [DBG] pgmap v162: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 3 op/s 2021-11-20T06:03:51.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:51 smithi139 conmon[33501]: cluster 2021-11-20T06:03:50.213413+0000 mgr.smithi139.dplyxn (mgr.14162) 201 : cluster [DBG] pgmap v162: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 3 op/s 2021-11-20T06:03:53.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:53 smithi178 conmon[36600]: cluster 2021-11-20T06:03:52.213998+0000 mgr.smithi139.dplyxn (mgr.14162) 202 : cluster [DBG] pgmap v163: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2021-11-20T06:03:53.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:53 smithi178 conmon[36600]: 2021-11-20T06:03:53.475 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:53 smithi139 conmon[33501]: cluster 2021-11-20T06:03:52.213998+0000 mgr.smithi139.dplyxn (mgr.14162) 202 : cluster [DBG] pgmap v163: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2021-11-20T06:03:53.476 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:53 smithi139 conmon[33501]: 2021-11-20T06:03:54.883 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:54.897+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:54.883 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:54.897+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 6.400000 2021-11-20T06:03:54.883 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:03:54.897+0000 7f2575ffb700 1 client.14534 ms_handle_refused on v2:172.21.15.178:6824/1867304351 2021-11-20T06:03:55.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:55 smithi139 conmon[33501]: cluster 2021-11-20T06:03:54.214692+0000 mgr.smithi139.dplyxn (mgr.14162) 203 : cluster [DBG] pgmap v164: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2021-11-20T06:03:55.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:55 smithi178 conmon[36600]: cluster 2021-11-20T06:03:54.214692+0000 mgr.smithi139.dplyxn (mgr.14162) 203 : cluster [DBG] pgmap v164: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2021-11-20T06:03:57.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:57 smithi139 conmon[33501]: cluster 2021-11-20T06:03:56.215380+0000 mgr.smithi139.dplyxn (mgr.14162) 204 : cluster [DBG] pgmap v165: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2021-11-20T06:03:57.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:57 smithi178 conmon[36600]: cluster 2021-11-20T06:03:56.215380+0000 mgr.smithi139.dplyxn (mgr.14162) 204 : cluster [DBG] 2021-11-20T06:03:57.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:57 smithi178 conmon[36600]: pgmap v165: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2021-11-20T06:03:59.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:59 smithi139 conmon[33501]: cluster 2021-11-20T06:03:58.215850+0000 mgr.smithi139.dplyxn (mgr.14162) 205 : 2021-11-20T06:03:59.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:03:59 smithi139 conmon[33501]: cluster [DBG] pgmap v166: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2021-11-20T06:03:59.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:03:59 smithi178 conmon[36600]: cluster 2021-11-20T06:03:58.215850+0000 mgr.smithi139.dplyxn (mgr.14162) 205 : cluster [DBG] pgmap v166: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2021-11-20T06:04:01.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:01.300+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:04:01.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:01.301+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T06:04:01.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:01.301+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T06:04:01.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:01.301+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T06:04:01.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:01.301+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 12.800000 2021-11-20T06:04:01.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:01 smithi139 conmon[33501]: cluster 2021-11-20T06:04:00.216413+0000 mgr.smithi139.dplyxn (mgr.14162) 206 : 2021-11-20T06:04:01.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:01 smithi139 conmon[33501]: cluster [DBG] pgmap v167: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2021-11-20T06:04:01.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:01 smithi178 conmon[36600]: cluster 2021-11-20T06:04:00.216413+0000 mgr.smithi139.dplyxn (mgr.14162) 206 : cluster [DBG] pgmap v167: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2021-11-20T06:04:02.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:02 smithi178 conmon[36600]: cluster 2021-11-20T06:04:02.184862+0000 mon.smithi139 (mon.0) 543 : cluster [WRN] Replacing daemon mds.cephfs.smithi178.blvkbx as rank 0 with standby daemon mds.cephfs.smithi139.riczqb 2021-11-20T06:04:02.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:02 smithi178 conmon[36600]: cluster 2021-11-20T06:04:02.184916+0000 mon.smithi139 (mon.0) 544 : cluster [INF] MDS daemon mds.cephfs.smithi178.blvkbx is removed because it is dead or otherwise unavailable. 2021-11-20T06:04:02.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:02 smithi178 conmon[36600]: cluster 2021-11-20T06:04:02. 2021-11-20T06:04:02.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:02 smithi178 conmon[36600]: 185508+0000 mon.smithi139 (mon.0) 545 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2021-11-20T06:04:02.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:02 smithi178 conmon[36600]: cluster 2021-11-20T06 2021-11-20T06:04:02.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:02 smithi178 conmon[36600]: :04:02.189839+0000 mon.smithi139 (mon.0) 546 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2021-11-20T06:04:02.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:02 smithi178 conmon[36600]: cluster 2021-11-20T 2021-11-20T06:04:02.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:02 smithi178 conmon[36600]: 06:04:02.193774+0000 mon.smithi139 (mon.0) 547 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:replay} 2 up:standby 2021-11-20T06:04:02.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:02 smithi139 conmon[33501]: cluster 2021-11-20T06:04:02.184862+0000 mon.smithi139 (mon.0) 543 : cluster [WRN] Replacing daemon mds.cephfs.smithi178.blvkbx as rank 0 with standby daemon mds.cephfs.smithi139.riczqb 2021-11-20T06:04:02.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:02 smithi139 conmon[33501]: cluster 2021-11-20T06:04:02.184916+0000 mon.smithi139 (mon.0) 544 : cluster [INF] MDS daemon mds.cephfs.smithi178.blvkbx is removed because it is dead or otherwise unavailable. 2021-11-20T06:04:02.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:02 smithi139 conmon[33501]: cluster 2021-11-20T06:04:02.185508+0000 mon.smithi139 (mon.0) 545 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2021-11-20T06:04:02.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:02 smithi139 conmon[33501]: cluster 2021-11-20T06:04: 2021-11-20T06:04:02.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:02 smithi139 conmon[33501]: 02.189839+0000 mon.smithi139 (mon.0) 546 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2021-11-20T06:04:02.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:02 smithi139 conmon[33501]: cluster 2021-11-20T06:04:02.193774+0000 mon.smithi139 (mon.0) 547 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:replay} 2 up:standby 2021-11-20T06:04:03.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:03 smithi139 conmon[33501]: cluster 2021-11-20T06:04:02.216836+0000 mgr.smithi139.dplyxn (mgr.14162) 207 : cluster [DBG] pgmap v169: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2021-11-20T06:04:03.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:03 smithi139 conmon[33501]: cluster 2021-11-20T06:04:03.195913+0000 mon.smithi139 (mon.0) 548 : cluster [DBG] mds.? [v2:172.21.15.178:6824/3308879813,v1:172.21.15.178:6825/3308879813] up:boot 2021-11-20T06:04:03.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:03 smithi139 conmon[33501]: cluster 2021-11-20T06:04:03.195979+0000 mon.smithi139 ( 2021-11-20T06:04:03.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:03 smithi139 conmon[33501]: mon.0) 549 : cluster [DBG] mds.? [v2:172.21.15.139:6826/724620583,v1:172.21.15.139:6827/724620583] up:reconnect 2021-11-20T06:04:03.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:03 smithi139 conmon[33501]: cluster 2021-11-20T06:04:03.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:03 smithi139 conmon[33501]: 2021-11-20T06:04:03.196032+0000 mon.smithi139 (mon.0) 2021-11-20T06:04:03.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:03 smithi139 conmon[33501]: 550 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:reconnect} 3 up:standby 2021-11-20T06:04:03.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:03 smithi139 conmon[33501]: audit 2021-11-20T 2021-11-20T06:04:03.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:03 smithi139 conmon[33501]: 06:04:03.196171+0000 mon.smithi139 (mon.0 2021-11-20T06:04:03.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:03 smithi139 conmon[33501]: ) 551 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.blvkbx"}]: dispatch 2021-11-20T06:04:03.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:03 smithi178 conmon[36600]: cluster 2021-11-20T06:04:02.216836+0000 mgr.smithi139.dplyxn (mgr.14162) 207 : cluster [DBG] pgmap v169: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2021-11-20T06:04:03.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:03 smithi178 conmon[36600]: cluster 2021-11-20T06:04:03.195913+0000 mon.smithi139 (mon.0 2021-11-20T06:04:03.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:03 smithi178 conmon[36600]: ) 548 : cluster [DBG] mds.? [v2:172.21.15.178:6824/3308879813,v1:172.21.15.178:6825/3308879813] up:boot 2021-11-20T06:04:03.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:03 smithi178 conmon[36600]: cluster 2021-11-20T06:04:03.195979+0000 mon.smithi139 (mon.0) 549 : cluster [DBG] mds.? [v2:172.21.15.139:6826/724620583,v1:172.21.15.139:6827/724620583] up:reconnect 2021-11-20T06:04:03.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:03 smithi178 conmon[36600]: cluster 2021-11-20T06:04:03.196032+0000 mon.smithi139 (mon.0) 550 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:reconnect} 3 up:standby 2021-11-20T06:04:03.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:03 smithi178 conmon[36600]: audit 2021-11-20T06: 2021-11-20T06:04:03.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:03 smithi178 conmon[36600]: 04:03.196171+0000 mon.smithi139 (mon.0) 551 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.blvkbx"}]: dispatch 2021-11-20T06:04:04.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:04 smithi139 conmon[33501]: cluster 2021-11-20T06:04:03.215770+0000 mds.cephfs.smithi139.riczqb (mds 2021-11-20T06:04:04.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:04 smithi139 conmon[33501]: .0) 1 : cluster [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0229995 2021-11-20T06:04:04.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:04 smithi139 conmon[33501]: cluster 2021-11-20T06:04:04.202045+0000 mon.smithi139 (mon.0) 552 : cluster [DBG] mds.? [v2:172.21.15.139:6826/724620583,v1:172.21.15.139:6827/724620583] up:rejoin 2021-11-20T06:04:04.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:04 smithi139 conmon[33501]: 2021-11-20T06:04:04.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:04 smithi139 conmon[33501]: cluster 2021-11-20 2021-11-20T06:04:04.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:04 smithi139 conmon[33501]: T06:04:04.202108+0000 mon.smithi139 (mon.0) 553 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:04:04.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:04 smithi178 conmon[36600]: cluster 2021-11-20T06:04:03.215770+0000 mds.cephfs.smithi139.riczqb (mds.0) 1 : cluster [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0229995 2021-11-20T06:04:04.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:04 smithi178 conmon[36600]: cluster 2021-11-20T06:04:04.202045+0000 mon.smithi139 (mon.0) 552 : cluster [DBG] mds.? [v2:172.21.15.139:6826/724620583,v1:172.21.15.139:6827/724620583] up:rejoin 2021-11-20T06:04:04.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:04 smithi178 conmon[36600]: cluster 2021-11-20T06: 2021-11-20T06:04:04.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:04 smithi178 conmon[36600]: 04:04.202108+0000 mon.smithi139 (mon.0) 553 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:04:05.460 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:05 smithi139 conmon[33501]: cluster 2021-11-20T06:04:04.217294+0000 mgr.smithi139.dplyxn (mgr.14162) 208 : cluster [DBG] pgmap v170: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2021-11-20T06:04:05.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:05 smithi178 conmon[36600]: cluster 2021-11-20T06:04:04.217294+0000 mgr.smithi139.dplyxn (mgr.14162) 208 : cluster [DBG] pgmap v170: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2021-11-20T06:04:07.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:07 smithi139 conmon[33501]: cluster 2021-11-20T06:04:06.217871+0000 mgr.smithi139.dplyxn (mgr.14162) 209 : cluster [DBG] pgmap v171: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2021-11-20T06:04:07.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:07 smithi139 conmon[33501]: cluster 2021-11-20T06:04:06.295351 2021-11-20T06:04:07.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:07 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 554 : cluster [DBG] mds.? [v2:172.21.15.178:6824/3308879813,v1:172.21.15.178:6825/3308879813] up:standby 2021-11-20T06:04:07.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:07 smithi139 conmon[33501]: cluster 2021-11-20T06:04:06.295410+0000 mon.smithi139 (mon.0) 555 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:04:07.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:07 smithi178 conmon[36600]: cluster 2021-11-20T06:04:06.217871+0000 mgr.smithi139.dplyxn (mgr.14162) 209 : cluster [DBG] pgmap v171: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2021-11-20T06:04:07.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:07 smithi178 conmon[36600]: cluster 2021-11- 2021-11-20T06:04:07.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:07 smithi178 conmon[36600]: 20T06:04:06.295351+0000 mon.smithi139 (mon.0) 554 : cluster [DBG] mds.? [v2:172.21.15.178:6824/3308879813,v1:172.21.15.178:6825/3308879813] up:standby 2021-11-20T06:04:07.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:07 smithi178 conmon[36600]: cluster 2021-11-20T06:04:06.295410+0000 mon.smithi139 (mon.0) 555 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:04:09.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:09 smithi139 conmon[33501]: cluster 2021-11-20T06:04:08.218526+0000 mgr.smithi139.dplyxn (mgr.14162) 210 : cluster [DBG] pgmap v172: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2021-11-20T06:04:09.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:09 smithi178 conmon[36600]: cluster 2021-11-20T06:04:08.218526+0000 mgr.smithi139.dplyxn (mgr.14162) 210 : cluster [DBG] pgmap v172: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2021-11-20T06:04:09.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:09 smithi178 conmon[36600]: 2021-11-20T06:04:11.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:11 smithi178 conmon[36600]: cluster 2021-11-20T06:04:10.219005+0000 mgr.smithi139.dplyxn (mgr.14162) 211 : cluster [DBG] 2021-11-20T06:04:11.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:11 smithi178 conmon[36600]: pgmap v173: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 1 op/s 2021-11-20T06:04:11.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:11 smithi139 conmon[33501]: cluster 2021-11-20T06:04:10.219005+0000 mgr.smithi139.dplyxn (mgr.14162) 211 : cluster [DBG] pgmap v173: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 1 op/s 2021-11-20T06:04:13.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:13 smithi178 conmon[36600]: cluster 2021-11-20T06:04:12.219737+0000 mgr.smithi139.dplyxn (mgr.14162) 212 : cluster [DBG] pgmap v174: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 816 B/s rd, 1 op/s 2021-11-20T06:04:13.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:13 smithi139 conmon[33501]: cluster 2021-11-20T06:04:12.219737+0000 mgr.smithi139.dplyxn (mgr.14162) 212 : cluster [DBG] pgmap v174: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 816 B/s rd, 1 op/s 2021-11-20T06:04:14.089 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:14.104+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:04:14.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:14.104+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T06:04:14.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:14.105+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T06:04:14.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:14.105+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T06:04:14.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:14.105+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T06:04:15.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:15 smithi139 conmon[33501]: cluster 2021-11-20T06:04:14.220242+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:04:15.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:15 smithi139 conmon[33501]: 14162) 213 : cluster [DBG] pgmap v175: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 1 op/s 2021-11-20T06:04:15.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:15 smithi178 conmon[36600]: cluster 2021-11-20T06:04:14.220242+0000 mgr.smithi139.dplyxn (mgr.14162) 213 : cluster [DBG] pgmap v175: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 1 op/s 2021-11-20T06:04:16.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:16 smithi139 conmon[33501]: debug 2021-11-20T06:04:16.274+0000 7f2a21654700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:04:16.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:16 smithi139 conmon[33501]: cluster 2021-11-20T06: 2021-11-20T06:04:16.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:16 smithi139 conmon[33501]: 04:15.301069+0000 mon.smithi139 (mon.0) 556 : cluster [INF] daemon mds.cephfs.smithi139.riczqb restarted 2021-11-20T06:04:16.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:16 smithi139 conmon[33501]: cluster 2021 2021-11-20T06:04:16.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:16 smithi139 conmon[33501]: -11-20T06:04:15.305004+0000 mon.smithi139 (mon.0) 557 : cluster [DBG] 2021-11-20T06:04:16.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:16 smithi139 conmon[33501]: osdmap e43: 6 total, 6 up, 6 in 2021-11-20T06:04:16.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:16 smithi178 conmon[36600]: cluster 2021-11-20T06:04:15.301069+0000 mon.smithi139 (mon.0) 556 : cluster [INF] daemon mds.cephfs.smithi139.riczqb restarted 2021-11-20T06:04:16.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:16 smithi178 conmon[36600]: cluster 2021-11-20T06:04:15.305004+0000 mon.smithi139 (mon.0) 557 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2021-11-20T06:04:17.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: cluster 2021-11-20T06:04:16.220693+0000 mgr.smithi139.dplyxn (mgr.14162) 214 : cluster [DBG] pgmap v177: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-20T06:04:17.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: cluster 2021-11-20T06:04:16.274905+0000 mon.smithi139 (mon.0) 558 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:04:17.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: cluster 2021-11-20T06:04:16.281984+0000 mon.smithi139 (mon.0) 559 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1489716627,v1:172.21.15.139:6827/1489716627] up:boot 2021-11-20T06:04:17.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: cluster 2021-11-20T06:04:16.282055+0000 mon.smithi139 (mon.0) 560 : cluster [INF] Standby daemon mds.cephfs.smithi139.oudezh assigned to filesystem cephfs as rank 0 2021-11-20T06:04:17.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: cluster 2021- 2021-11-20T06:04:17.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: 11-20T06:04:16.282188+0000 mon.smithi139 (mon.0) 561 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:04:17.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: cluster 2021-11-20T06:04:17.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: 2021-11-20T06:04: 2021-11-20T06:04:17.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: 16.282452+0000 mon.smithi139 (mon.0) 2021-11-20T06:04:17.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: 562 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T06:04:17.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: audit 2021- 2021-11-20T06:04:17.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: 11-20T06:04:16.282576+0000 2021-11-20T06:04:17.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: mon.smithi139 (mon.0) 563 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.riczqb"}]: dispatch 2021-11-20T06:04:17.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: 2021-11-20T06:04:17.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: cluster 2021-11-20T06:04:17.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: 2021-11-20T06:04:16.285867+0000 mon.smithi139 2021-11-20T06:04:17.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: (mon.0) 564 : cluster [DBG] 2021-11-20T06:04:17.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:17 smithi139 conmon[33501]: fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:replay} 3 up:standby 2021-11-20T06:04:17.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: cluster 2021-11-20T06:04:16.220693+0000 mgr.smithi139.dplyxn (mgr.14162) 214 : cluster [DBG] pgmap v177: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-20T06:04:17.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: cluster 2021-11-20T06:04:16.274905+0000 mon.smithi139 (mon.0) 558 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:04:17.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: cluster 2021-11-20T06:04:16.281984+0000 mon.smithi139 (mon.0) 559 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1489716627,v1:172.21.15.139:6827/1489716627] up:boot 2021-11-20T06:04:17.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: cluster 2021-11-20T06:04:16.282055+0000 mon.smithi139 (mon.0) 560 : cluster 2021-11-20T06:04:17.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: [INF] Standby daemon mds.cephfs.smithi139.oudezh assigned to filesystem cephfs as rank 0 2021-11-20T06:04:17.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: cluster 2021-11-20T06:04:16.282188 2021-11-20T06:04:17.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: +0000 mon.smithi139 (mon.0) 561 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:04:17.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: 2021-11-20T06:04:17.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: cluster 2021- 2021-11-20T06:04:17.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: 11-20T06:04:16.282452+0000 mon.smithi139 (mon.0) 2021-11-20T06:04:17.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: 562 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T06:04:17.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: audit 2021-11-20T06: 2021-11-20T06:04:17.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: 04:16.282576+0000 mon.smithi139 (mon.0) 563 2021-11-20T06:04:17.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.riczqb"}]: dispatch 2021-11-20T06:04:17.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: cluster 2021-11-20T06:04:17.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: 2021-11-20T06:04:16.285867+0000 mon.smithi139 (mon.0 2021-11-20T06:04:17.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:17 smithi178 conmon[36600]: ) 564 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:replay} 3 up:standby 2021-11-20T06:04:18.509 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:18 smithi139 conmon[33501]: cluster 2021-11-20T06:04:17.286516+0000 mon.smithi139 (mon.0 2021-11-20T06:04:18.509 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:18 smithi139 conmon[33501]: ) 565 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2967963544,v1:172.21.15.139:6829/2967963544] up:reconnect 2021-11-20T06:04:18.509 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:18 smithi139 conmon[33501]: cluster 2021-11-20T06:04:17.286585+0000 mon.smithi139 (mon.0) 566 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:reconnect} 3 up:standby 2021-11-20T06:04:18.510 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:18 smithi139 conmon[33501]: cluster 2021-11-20T06:04:17.306806+0000 mds.cephfs.smithi139.oudezh (mds.0) 1 : cluster [DBG] 2021-11-20T06:04:18.510 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:18 smithi139 conmon[33501]: reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0219995 2021-11-20T06:04:18.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:18 smithi178 conmon[36600]: cluster 2021-11-20T06:04:17.286516+0000 mon.smithi139 (mon.0) 565 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2967963544,v1:172.21.15.139:6829/2967963544] up:reconnect 2021-11-20T06:04:18.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:18 smithi178 conmon[36600]: cluster 2021-11-20T06:04:17.286585+0000 mon.smithi139 (mon.0) 566 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:reconnect} 3 up:standby 2021-11-20T06:04:18.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:18 smithi178 conmon[36600]: cluster 2021-11-20T06:04:17.306806+0000 mds.cephfs.smithi139.oudezh (mds.0) 1 : cluster [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0219995 2021-11-20T06:04:19.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:19 smithi139 conmon[33501]: cluster 2021-11-20T06:04:18.221211+0000 mgr.smithi139.dplyxn (mgr.14162) 215 : cluster [DBG] pgmap v178: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T06:04:19.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:19 smithi139 conmon[33501]: cluster 2021-11-20T06:04:18.288224+0000 mon.smithi139 (mon.0) 567 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2967963544,v1:172.21.15.139:6829/2967963544] up:rejoin 2021-11-20T06:04:19.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:19 smithi139 conmon[33501]: cluster 2021-11-20T06:04:18.288274+0000 mon.smithi139 (mon.0) 568 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:04:19.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:19 smithi178 conmon[36600]: cluster 2021-11-20T06:04:18.221211+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:04:19.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:19 smithi178 conmon[36600]: 14162) 215 : cluster [DBG] pgmap v178: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T06:04:19.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:19 smithi178 conmon[36600]: cluster 2021-11-20T06:04:18.288224+0000 mon.smithi139 (mon.0) 567 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2967963544,v1:172.21.15.139:6829/2967963544] up:rejoin 2021-11-20T06:04:19.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:19 smithi178 conmon[36600]: cluster 2021-11-20T06:04:19.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:19 smithi178 conmon[36600]: 2021-11-20T06:04:18.288274+0000 mon.smithi139 (mon.0) 568 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:04:21.701 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:21 smithi178 conmon[36600]: cluster 2021-11-20T06:04:20.221726+0000 mgr.smithi139.dplyxn (mgr.14162) 216 : cluster [DBG] pgmap v179: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:04:21.702 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:21 smithi178 conmon[36600]: cluster 2021-11-20T06:04:20.297698+0000 mon.smithi139 (mon.0) 569 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1489716627,v1:172.21.15.139:6827/1489716627] up:standby 2021-11-20T06:04:21.702 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:21 smithi178 conmon[36600]: cluster 2021-11-20T06:04:20.297761+0000 mon.smithi139 (mon.0) 570 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:04:21.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:21 smithi139 conmon[33501]: cluster 2021-11-20T06:04:20.221726+0000 mgr.smithi139.dplyxn (mgr.14162) 216 : cluster [DBG] pgmap v179: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:04:21.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:21 smithi139 conmon[33501]: cluster 2021-11-20T06:04:20.297698+0000 2021-11-20T06:04:21.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:21 smithi139 conmon[33501]: mon.smithi139 (mon.0) 569 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1489716627,v1:172.21.15.139:6827/1489716627] up:standby 2021-11-20T06:04:21.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:21 smithi139 conmon[33501]: cluster 2021-11-20T06:04:20.297761+0000 mon.smithi139 (mon.0) 570 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:04:23.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:23 smithi178 conmon[36600]: cluster 2021-11-20T06:04:22.222113+0000 mgr.smithi139.dplyxn (mgr.14162) 217 : cluster [DBG] 2021-11-20T06:04:23.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:23 smithi178 conmon[36600]: pgmap v180: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:04:23.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:23 smithi139 conmon[33501]: cluster 2021-11-20T06:04:22.222113+0000 mgr.smithi139.dplyxn (mgr.14162) 217 : cluster [DBG] 2021-11-20T06:04:23.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:23 smithi139 conmon[33501]: pgmap v180: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:04:25.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:25 smithi139 conmon[33501]: cluster 2021-11-20T06:04:24.222579+0000 mgr.smithi139.dplyxn (mgr.14162) 218 : cluster [DBG] pgmap v181: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:04:25.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:25 smithi178 conmon[36600]: cluster 2021-11-20T06:04:24.222579+0000 mgr.smithi139.dplyxn (mgr.14162) 218 : cluster [DBG] pgmap v181: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:04:27.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:27 smithi139 conmon[33501]: cluster 2021-11-20T06:04:26.223032+0000 mgr.smithi139.dplyxn (mgr.14162) 219 : cluster [DBG] pgmap v182: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:04:27.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:27 smithi178 conmon[36600]: cluster 2021-11-20T06:04:26.223032+0000 mgr.smithi139.dplyxn (mgr.14162) 219 : 2021-11-20T06:04:27.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:27 smithi178 conmon[36600]: cluster [DBG] pgmap v182: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:04:29.099 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:29.113+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:04:29.100 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:29.113+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T06:04:29.100 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:29.113+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T06:04:29.100 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:29.113+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T06:04:29.100 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:29.114+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T06:04:29.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:29 smithi139 conmon[33501]: cluster 2021-11-20T06:04:28.223565+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:04:29.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:29 smithi139 conmon[33501]: 14162) 220 : cluster [DBG] pgmap v183: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:04:29.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:29 smithi178 conmon[36600]: cluster 2021-11-20T06:04:28.223565+0000 mgr.smithi139.dplyxn (mgr.14162) 220 : cluster [DBG] pgmap v183: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:04:31.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:31 smithi139 conmon[33501]: cluster 2021-11-20T06:04:30.223967+0000 mgr.smithi139.dplyxn (mgr.14162) 221 : cluster 2021-11-20T06:04:31.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:31 smithi139 conmon[33501]: [DBG] pgmap v184: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 0 op/s 2021-11-20T06:04:31.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:31 smithi139 conmon[33501]: audit 2021-11-20T06:04:31.253496+0000 mon.smithi139 (mon.0) 571 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:04:31.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:31 smithi139 conmon[33501]: audit 2021-11-20T06:04: 2021-11-20T06:04:31.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:31 smithi139 conmon[33501]: 31.253906+0000 mon.smithi139 (mon.0) 572 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:04:31.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:31 smithi178 conmon[36600]: cluster 2021-11-20T06:04:30.223967+0000 mgr.smithi139.dplyxn (mgr.14162) 221 : cluster [DBG] 2021-11-20T06:04:31.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:31 smithi178 conmon[36600]: pgmap v184: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 0 op/s 2021-11-20T06:04:31.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:31 smithi178 conmon[36600]: audit 2021-11-20T06:04:31.253496+0000 mon.smithi139 (mon.0) 571 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:04:31.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:31 smithi178 conmon[36600]: audit 2021-11-20T06:04:31.253906+0000 mon.smithi139 (mon.0) 572 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:04:33.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:33 smithi178 conmon[36600]: cluster 2021-11-20T06:04:32.224432+0000 mgr.smithi139.dplyxn (mgr.14162) 222 : cluster [DBG] pgmap v185: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 85 B/s rd, 0 op/s 2021-11-20T06:04:33.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:33 smithi139 conmon[33501]: cluster 2021-11-20T06:04:32.224432+0000 mgr.smithi139.dplyxn (mgr.14162) 222 : cluster [DBG] pgmap v185: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 85 B/s rd, 0 op/s 2021-11-20T06:04:33.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:33 smithi139 conmon[33501]: 2021-11-20T06:04:34.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:34 smithi178 conmon[36600]: cluster 2021-11-20T06:04:33.267234+0000 mon.smithi139 (mon.0) 573 : cluster [INF] daemon mds.cephfs.smithi139.oudezh restarted 2021-11-20T06:04:34.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:34 smithi178 conmon[36600]: 2021-11-20T06:04:34.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:34 smithi178 conmon[36600]: cluster 2021-11-20T06:04:33.273084+0000 mon.smithi139 (mon.0) 574 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2021-11-20T06:04:34.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:34 smithi139 conmon[33501]: debug 2021-11-20T06:04:34.199+0000 7f2a21654700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:04:34.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:34 smithi139 conmon[33501]: cluster 2021-11-20T06:04:33.267234+0000 mon.smithi139 (mon.0) 573 : cluster [INF] daemon mds.cephfs.smithi139.oudezh restarted 2021-11-20T06:04:34.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:34 smithi139 conmon[33501]: cluster 2021-11-20T06:04:33.273084+0000 mon.smithi139 (mon.0) 574 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2021-11-20T06:04:35.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:35 smithi178 conmon[36600]: cluster 2021-11-20T06:04:34.200300+0000 mon.smithi139 (mon.0) 575 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:04:35.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:35 smithi178 conmon[36600]: 2021-11-20T06:04:35.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:35 smithi178 conmon[36600]: cluster 2021-11-20T06:04:34.210085+0000 mon.smithi139 (mon.0) 576 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2673475897,v1:172.21.15.139:6829/2673475897] up:boot 2021-11-20T06:04:35.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:35 smithi178 conmon[36600]: cluster 2021-11-20T06:04:34.210150+0000 mon.smithi139 (mon.0) 577 : cluster [INF] Standby daemon mds.cephfs.smithi139.riczqb assigned to filesystem cephfs as rank 0 2021-11-20T06:04:35.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:35 smithi178 conmon[36600]: cluster 2021-11-20T06:04:34.210259+0000 mon.smithi139 (mon.0) 578 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:04:35.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:35 smithi178 conmon[36600]: cluster 2021-11-20T06:04 2021-11-20T06:04:35.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:35 smithi178 conmon[36600]: :34.210503+0000 mon.smithi139 (mon.0) 579 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T06:04:35.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:35 smithi178 conmon[36600]: audit 2021-11 2021-11-20T06:04:35.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:35 smithi178 conmon[36600]: -20T06:04:34.210625+0000 mon.smithi139 (mon.0) 580 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.oudezh"}]: dispatch 2021-11-20T06:04:35.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:35 smithi178 conmon[36600]: cluster 2021-11-20T06:04:34.214519+0000 mon.smithi139 (mon.0) 581 : 2021-11-20T06:04:35.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:35 smithi178 conmon[36600]: cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:replay} 3 up:standby 2021-11-20T06:04:35.445 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:35 smithi178 conmon[36600]: cluster 2021-11-20T06:04:35.446 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:35 smithi178 conmon[36600]: 2021-11-20T06:04:34.224789+0000 mgr.smithi139.dplyxn (mgr.14162) 223 : cluster [DBG] pgmap v187: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:04:35.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:35 smithi139 conmon[33501]: cluster 2021-11-20T06:04:34.200300+0000 mon.smithi139 (mon.0) 575 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:04:35.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:35 smithi139 conmon[33501]: cluster 2021-11-20T06:04:34.210085+0000 mon.smithi139 (mon.0) 576 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2673475897,v1:172.21.15.139:6829/2673475897] up:boot 2021-11-20T06:04:35.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:35 smithi139 conmon[33501]: cluster 2021-11-20T06:04:34.210150+0000 mon.smithi139 (mon.0) 577 : cluster [INF] Standby daemon mds.cephfs.smithi139.riczqb assigned to filesystem cephfs as rank 0 2021-11-20T06:04:35.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:35 smithi139 conmon[33501]: cluster 2021-11-20T06:04:34.210259+0000 mon.smithi139 (mon.0) 578 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:04:35.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:35 smithi139 conmon[33501]: cluster 2021-11-20T06:04:35.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:35 smithi139 conmon[33501]: 2021-11-20T06:04:34.210503+0000 mon.smithi139 (mon.0) 579 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T06:04:35.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:35 smithi139 conmon[33501]: audit 2021-11-20T06 2021-11-20T06:04:35.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:35 smithi139 conmon[33501]: :04:34.210625+0000 mon.smithi139 (mon.0) 580 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.oudezh"}]: dispatch 2021-11-20T06:04:35.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:35 smithi139 conmon[33501]: cluster 2021-11-20 2021-11-20T06:04:35.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:35 smithi139 conmon[33501]: T06:04:34.214519+0000 mon.smithi139 (mon.0) 581 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:replay} 3 up:standby 2021-11-20T06:04:35.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:35 smithi139 conmon[33501]: cluster 2021-11-20T06:04:34.224789+0000 mgr.smithi139.dplyxn (mgr.14162) 223 2021-11-20T06:04:35.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:35 smithi139 conmon[33501]: : cluster [DBG] pgmap v187: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:04:36.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:36 smithi139 conmon[33501]: cluster 2021-11-20T06:04:35.222120+0000 mon.smithi139 (mon.0) 582 : cluster [DBG] 2021-11-20T06:04:36.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:36 smithi139 conmon[33501]: mds.? [v2:172.21.15.139:6826/1489716627,v1:172.21.15.139:6827/1489716627] up:reconnect 2021-11-20T06:04:36.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:36 smithi139 conmon[33501]: cluster 2021-11-20T06:04:35.222191+0000 mon.smithi139 (mon.0) 583 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:reconnect} 3 up:standby 2021-11-20T06:04:36.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:36 smithi139 conmon[33501]: cluster 2021-11-20T06:04:35.243969+0000 mds.cephfs.smithi139.riczqb ( 2021-11-20T06:04:36.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:36 smithi139 conmon[33501]: mds.0) 1 : cluster [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0239995 2021-11-20T06:04:36.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:36 smithi178 conmon[36600]: cluster 2021-11-20T06:04:35.222120+0000 mon.smithi139 (mon.0) 582 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1489716627,v1:172.21.15.139:6827/1489716627] up:reconnect 2021-11-20T06:04:36.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:36 smithi178 conmon[36600]: cluster 2021-11-20T06:04:35.222191+0000 mon.smithi139 (mon.0) 583 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:reconnect} 3 up:standby 2021-11-20T06:04:36.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:36 smithi178 conmon[36600]: cluster 2021-11-20T06:04:35.243969+0000 mds.cephfs.smithi139.riczqb (mds.0) 1 : cluster [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0239995 2021-11-20T06:04:37.460 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:37 smithi139 conmon[33501]: cluster 2021-11-20T06:04:36.224311+0000 mon.smithi139 (mon.0) 584 : cluster [DBG] 2021-11-20T06:04:37.460 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:37 smithi139 conmon[33501]: mds.? [v2:172.21.15.139:6826/1489716627,v1:172.21.15.139:6827/1489716627] up:rejoin 2021-11-20T06:04:37.461 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:37 smithi139 conmon[33501]: cluster 2021-11-20T06:04:36.224381+0000 mon.smithi139 (mon.0) 585 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:04:37.461 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:37 smithi139 conmon[33501]: cluster 2021-11-20T06:04:36.225229+0000 mgr.smithi139.dplyxn (mgr.14162) 224 : cluster [DBG] pgmap v188: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T06:04:37.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:37 smithi178 conmon[36600]: cluster 2021-11-20T06:04:36.224311+0000 mon.smithi139 (mon.0) 584 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1489716627,v1:172.21.15.139:6827/1489716627] up:rejoin 2021-11-20T06:04:37.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:37 smithi178 conmon[36600]: cluster 2021-11-20T06:04:36.224381+0000 mon.smithi139 (mon.0) 585 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:04:37.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:37 smithi178 conmon[36600]: cluster 2021-11-20T06:04:36.225229+0000 mgr.smithi139.dplyxn (mgr.14162) 224 : cluster [DBG] pgmap v188: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T06:04:39.430 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:39 smithi139 conmon[33501]: cluster 2021-11-20T06:04:38.225636+0000 mgr.smithi139.dplyxn (mgr.14162) 225 : cluster [DBG] pgmap v189: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:04:39.431 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:39 smithi139 conmon[33501]: cluster 2021-11-20T06:04:38.235604+0000 mon.smithi139 (mon.0) 586 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2673475897,v1:172.21.15.139:6829/2673475897] up:standby 2021-11-20T06:04:39.431 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:39 smithi139 conmon[33501]: cluster 2021-11-20T06:04:38.235664+0000 mon.smithi139 (mon.0) 587 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:04:39.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:39 smithi178 conmon[36600]: cluster 2021-11-20T06:04:38.225636+0000 mgr.smithi139.dplyxn (mgr.14162) 225 : cluster [DBG] pgmap v189: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:04:39.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:39 smithi178 conmon[36600]: cluster 2021-11-20T06:04:38.235604+0000 mon.smithi139 (mon.0) 586 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2673475897,v1:172.21.15.139:6829/2673475897] up:standby 2021-11-20T06:04:39.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:39 smithi178 conmon[36600]: cluster 2021-11-20T06:04:38. 2021-11-20T06:04:39.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:39 smithi178 conmon[36600]: 235664+0000 mon.smithi139 (mon.0) 587 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:04:41.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:41 smithi139 conmon[33501]: cluster 2021-11-20T06:04:40.226076+0000 mgr.smithi139.dplyxn (mgr.14162) 226 : cluster [DBG] pgmap v190: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:04:41.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:41 smithi178 conmon[36600]: cluster 2021-11-20T06:04:40.226076+0000 mgr.smithi139.dplyxn (mgr.14162) 226 : cluster [DBG] 2021-11-20T06:04:41.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:41 smithi178 conmon[36600]: pgmap v190: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:04:43.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:43 smithi178 conmon[36600]: cluster 2021-11-20T06:04:42.226554+0000 mgr.smithi139.dplyxn (mgr.14162) 227 2021-11-20T06:04:43.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:43 smithi178 conmon[36600]: : cluster [DBG] pgmap v191: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:04:43.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:43 smithi139 conmon[33501]: cluster 2021-11-20T06:04:42.226554+0000 mgr.smithi139.dplyxn (mgr.14162) 227 : cluster [DBG] pgmap v191: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:04:44.102 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:44.116+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:04:44.102 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:44.116+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T06:04:44.103 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:44.116+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T06:04:44.103 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:44.116+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T06:04:44.103 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:44.116+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T06:04:45.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:45 smithi139 conmon[33501]: cluster 2021-11-20T06:04:44.227081+0000 mgr.smithi139.dplyxn (mgr.14162) 228 2021-11-20T06:04:45.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:45 smithi139 conmon[33501]: : cluster [DBG] pgmap v192: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:04:45.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:45 smithi178 conmon[36600]: cluster 2021-11-20T06:04:44.227081+0000 mgr.smithi139.dplyxn (mgr.14162) 228 : cluster [DBG] pgmap v192: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:04:47.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:47 smithi139 conmon[33501]: cluster 2021-11-20T06:04:46.227602+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:04:47.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:47 smithi139 conmon[33501]: 229 : cluster [DBG] pgmap v193: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:04:47.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:47 smithi139 conmon[33501]: audit 2021-11-20T06:04:46.386375+0000 mon.smithi139 (mon.0) 588 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:04:47.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:47 smithi139 conmon[33501]: audit 2021-11-20T06:04:46.768832+0000 mon.smithi139 (mon 2021-11-20T06:04:47.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:47 smithi139 conmon[33501]: .0) 589 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:04:47.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:47 smithi139 conmon[33501]: cluster 2021-11-20T06:04:47. 2021-11-20T06:04:47.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:47 smithi139 conmon[33501]: 273677+0000 mon.smithi139 (mon.0) 590 : cluster [INF] daemon mds.cephfs.smithi139.riczqb restarted 2021-11-20T06:04:47.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:47 smithi139 conmon[33501]: cluster 2021-11- 2021-11-20T06:04:47.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:47 smithi139 conmon[33501]: 20T06:04:47.277818+0000 mon.smithi139 (mon.0) 591 2021-11-20T06:04:47.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:47 smithi139 conmon[33501]: : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2021-11-20T06:04:47.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:47 smithi178 conmon[36600]: cluster 2021-11-20T06:04:46.227602+0000 mgr.smithi139.dplyxn (mgr.14162) 229 : cluster [DBG] pgmap v193: 65 pgs: 65 active+clean; 2.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:04:47.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:47 smithi178 conmon[36600]: audit 2021-11-20T06:04:46.386375+0000 mon.smithi139 (mon.0) 588 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:04:47.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:47 smithi178 conmon[36600]: audit 2021-11-20T06:04:46.768832+0000 mon.smithi139 (mon.0) 589 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:04:47.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:47 smithi178 conmon[36600]: cluster 2021-11-20T06:04:47.273677+0000 mon.smithi139 (mon.0) 590 : cluster [INF] daemon mds.cephfs.smithi139.riczqb restarted 2021-11-20T06:04:47.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:47 smithi178 conmon[36600]: cluster 2021-11-20T06:04:47.277818+0000 mon.smithi139 (mon.0) 591 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2021-11-20T06:04:48.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: debug 2021-11-20T06:04:48.194+0000 7f2a21654700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:04:48.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: cluster 2021-11-20T06:04:48.195818+0000 mon.smithi139 (mon.0) 592 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:04:48.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: cluster 2021-11-20T06:04:48.199364+0000 mon.smithi139 2021-11-20T06:04:48.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: (mon.0) 593 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1581019423,v1:172.21.15.139:6827/1581019423] up:boot 2021-11-20T06:04:48.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: cluster 2021-11-20T06:04:48.199418+0000 mon.smithi139 (mon.0) 594 : cluster [INF] Standby daemon mds.cephfs.smithi139.oudezh assigned to filesystem cephfs as rank 0 2021-11-20T06:04:48.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: cluster 2021-11-20T06:04:48.199525 2021-11-20T06:04:48.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 595 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:04:48.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: cluster 2021- 2021-11-20T06:04:48.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: 11-20T06:04:48. 2021-11-20T06:04:48.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: 199785+0000 mon.smithi139 (mon.0) 596 2021-11-20T06:04:48.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T06:04:48.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:04:48.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: 20T06:04:48.200403+0000 mon.smithi139 2021-11-20T06:04:48.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: (mon.0) 597 : audit [DBG] 2021-11-20T06:04:48.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.riczqb"}]: dispatch 2021-11-20T06:04:48.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: cluster 2021- 2021-11-20T06:04:48.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: 11-20T06:04:48 2021-11-20T06:04:48.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: .204532+0000 mon.smithi139 (mon.0 2021-11-20T06:04:48.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:48 smithi139 conmon[33501]: ) 598 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:replay} 3 up:standby 2021-11-20T06:04:48.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: cluster 2021-11-20T06:04:48.195818+0000 mon.smithi139 (mon.0) 592 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:04:48.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: cluster 2021-11-20T06:04:48.199364+0000 mon.smithi139 (mon.0) 593 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1581019423,v1:172.21.15.139:6827/1581019423] up:boot 2021-11-20T06:04:48.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: cluster 2021-11-20T06:04:48.199418+0000 mon.smithi139 (mon 2021-11-20T06:04:48.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: .0) 594 : cluster [INF] Standby daemon mds.cephfs.smithi139.oudezh assigned to filesystem cephfs as rank 0 2021-11-20T06:04:48.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: cluster 2021 2021-11-20T06:04:48.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: -11-20T06:04:48.199525+0000 mon.smithi139 (mon. 2021-11-20T06:04:48.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: 0) 595 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:04:48.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: cluster 2021-11-20T06:04:48.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: 2021-11-20T06:04: 2021-11-20T06:04:48.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: 48.199785+0000 mon.smithi139 (mon.0) 596 : 2021-11-20T06:04:48.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T06:04:48.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: audit 2021-11- 2021-11-20T06:04:48.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: 20T06:04:48.200403+0000 2021-11-20T06:04:48.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: mon.smithi139 (mon.0) 597 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.riczqb"}]: dispatch 2021-11-20T06:04:48.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: cluster 2021 2021-11-20T06:04:48.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: -11-20T06:04:48. 2021-11-20T06:04:48.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: 204532+0000 mon.smithi139 (mon.0) 598 : cluster [DBG] 2021-11-20T06:04:48.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:48 smithi178 conmon[36600]: fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:replay} 3 up:standby 2021-11-20T06:04:49.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:49 smithi139 conmon[33501]: cluster 2021-11-20T06:04:48.227960+0000 mgr.smithi139.dplyxn (mgr.14162) 230 : cluster [DBG] pgmap v195: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T06:04:49.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:49 smithi139 conmon[33501]: cluster 2021-11-20T06:04:49.208638+0000 mon.smithi139 (mon.0) 599 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2673475897,v1:172.21.15.139:6829/2673475897] up:reconnect 2021-11-20T06:04:49.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:49 smithi139 conmon[33501]: cluster 2021-11-20T06:04:49.208715+0000 mon.smithi139 (mon.0) 600 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:reconnect} 3 up:standby 2021-11-20T06:04:49.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:49 smithi139 conmon[33501]: cluster 2021-11-20T06:04:49.228963+0000 mds.cephfs.smithi139.oudezh (mds 2021-11-20T06:04:49.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:49 smithi139 conmon[33501]: .0) 1 : cluster [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0219995 2021-11-20T06:04:49.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:49 smithi178 conmon[36600]: cluster 2021-11-20T06:04:48.227960+0000 mgr.smithi139.dplyxn (mgr.14162) 230 : cluster [DBG] pgmap v195: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T06:04:49.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:49 smithi178 conmon[36600]: cluster 2021-11-20T06:04:49.208638+0000 mon.smithi139 (mon.0) 599 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2673475897,v1:172.21.15.139:6829/2673475897] up:reconnect 2021-11-20T06:04:49.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:49 smithi178 conmon[36600]: cluster 2021-11- 2021-11-20T06:04:49.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:49 smithi178 conmon[36600]: 20T06:04:49.208715+0000 mon.smithi139 (mon.0) 600 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:reconnect} 3 up:standby 2021-11-20T06:04:49.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:49 smithi178 conmon[36600]: 2021-11-20T06:04:49.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:49 smithi178 conmon[36600]: cluster 2021-11-20T06 2021-11-20T06:04:49.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:49 smithi178 conmon[36600]: :04:49.228963+0000 mds.cephfs.smithi139.oudezh (mds 2021-11-20T06:04:49.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:49 smithi178 conmon[36600]: .0) 1 : cluster [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0219995 2021-11-20T06:04:51.476 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:51 smithi139 conmon[33501]: cluster 2021-11-20T06:04:50.210919+0000 mon.smithi139 (mon.0) 601 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2673475897,v1:172.21.15.139:6829/2673475897] up:rejoin 2021-11-20T06:04:51.476 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:51 smithi139 conmon[33501]: cluster 2021-11-20T06:04:50.210979+0000 mon.smithi139 (mon.0) 602 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:04:51.476 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:51 smithi139 conmon[33501]: cluster 2021-11-20T06:04:50.228549+0000 mgr.smithi139.dplyxn (mgr.14162) 231 : cluster [DBG] pgmap v196: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T06:04:51.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:51 smithi178 conmon[36600]: cluster 2021-11-20T06:04:50.210919+0000 mon.smithi139 (mon.0) 601 : cluster [DBG] 2021-11-20T06:04:51.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:51 smithi178 conmon[36600]: mds.? [v2:172.21.15.139:6828/2673475897,v1:172.21.15.139:6829/2673475897] up:rejoin 2021-11-20T06:04:51.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:51 smithi178 conmon[36600]: cluster 2021-11-20T06:04:50.210979+0000 mon.smithi139 (mon.0) 602 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:04:51.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:51 smithi178 conmon[36600]: cluster 2021-11 2021-11-20T06:04:51.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:51 smithi178 conmon[36600]: -20T06:04:50.228549+0000 mgr.smithi139.dplyxn (mgr.14162) 231 : cluster [DBG] pgmap v196: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T06:04:53.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:53 smithi139 conmon[33501]: cluster 2021-11-20T06:04:52.220675+0000 mon.smithi139 (mon.0) 603 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1581019423,v1:172.21.15.139:6827/1581019423] up:standby 2021-11-20T06:04:53.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:53 smithi139 conmon[33501]: cluster 2021-11-20T06:04:52.220756+0000 mon.smithi139 (mon.0) 604 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:04:53.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:53 smithi139 conmon[33501]: cluster 2021-11-20T06:04:52.228953 2021-11-20T06:04:53.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:53 smithi139 conmon[33501]: +0000 mgr.smithi139.dplyxn (mgr.14162) 232 : cluster [DBG] pgmap v197: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:04:53.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:53 smithi178 conmon[36600]: cluster 2021-11-20T06:04:52.220675+0000 mon.smithi139 (mon.0) 603 : cluster [DBG] 2021-11-20T06:04:53.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:53 smithi178 conmon[36600]: mds.? [v2:172.21.15.139:6826/1581019423,v1:172.21.15.139:6827/1581019423] up:standby 2021-11-20T06:04:53.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:53 smithi178 conmon[36600]: cluster 2021-11-20T06:04:52.220756+0000 mon.smithi139 (mon.0) 604 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:04:53.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:53 smithi178 conmon[36600]: cluster 2021-11-20T06:04:52.228953+0000 mgr.smithi139.dplyxn (mgr 2021-11-20T06:04:53.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:53 smithi178 conmon[36600]: .14162) 232 : cluster [DBG] pgmap v197: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:04:55.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:55 smithi139 conmon[33501]: cluster 2021-11-20T06:04:54.229370+0000 mgr.smithi139.dplyxn (mgr.14162) 233 : cluster 2021-11-20T06:04:55.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:55 smithi139 conmon[33501]: [DBG] pgmap v198: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:04:55.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:55 smithi178 conmon[36600]: cluster 2021-11-20T06:04:54.229370+0000 mgr.smithi139.dplyxn (mgr.14162) 233 : cluster [DBG] pgmap v198: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:04:57.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:57 smithi139 conmon[33501]: cluster 2021-11-20T06:04:56.229925+0000 mgr.smithi139.dplyxn (mgr.14162) 234 : cluster [DBG] pgmap v199: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:04:57.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:57 smithi178 conmon[36600]: cluster 2021-11-20T06:04:56.229925+0000 mgr.smithi139.dplyxn (mgr.14162) 234 : cluster [DBG] pgmap v199: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:04:59.104 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:59.118+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:04:59.104 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:59.118+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T06:04:59.104 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:59.118+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T06:04:59.105 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:59.118+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T06:04:59.105 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:04:59.118+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T06:04:59.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:59 smithi139 conmon[33501]: cluster 2021-11-20T06:04:58.230561+0000 mgr.smithi139.dplyxn (mgr.14162) 235 : cluster 2021-11-20T06:04:59.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:04:59 smithi139 conmon[33501]: [DBG] pgmap v200: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:04:59.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:04:59 smithi178 conmon[36600]: cluster 2021-11-20T06:04:58.230561+0000 mgr.smithi139.dplyxn (mgr.14162) 235 : cluster [DBG] pgmap v200: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:05:01.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:01 smithi139 conmon[33501]: cluster 2021-11-20T06:05:00.231136+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:05:01.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:01 smithi139 conmon[33501]: mgr.14162) 236 : cluster [DBG] pgmap v201: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:05:01.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:01 smithi178 conmon[36600]: cluster 2021-11-20T06:05:00.231136+0000 mgr.smithi139.dplyxn (mgr.14162) 236 : cluster [DBG] pgmap v201: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:05:03.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:03 smithi178 conmon[36600]: cluster 2021-11-20T06:05:02.231615+0000 mgr.smithi139.dplyxn (mgr.14162) 237 : cluster 2021-11-20T06:05:03.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:03 smithi178 conmon[36600]: [DBG] pgmap v202: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T06:05:03.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:03 smithi139 conmon[33501]: cluster 2021-11-20T06:05:02.231615+0000 mgr.smithi139.dplyxn (mgr.14162) 237 2021-11-20T06:05:03.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:03 smithi139 conmon[33501]: : cluster [DBG] pgmap v202: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T06:05:05.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:05 smithi139 conmon[33501]: debug 2021-11-20T06:05:05.284+0000 7f2a21654700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:05:05.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:05 smithi139 conmon[33501]: cluster 2021-11-20T06:05:04.232155+0000 mgr.smithi139.dplyxn 2021-11-20T06:05:05.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:05 smithi139 conmon[33501]: (mgr.14162) 238 : cluster [DBG] pgmap v203: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:05:05.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:05 smithi139 conmon[33501]: cluster 2021-11-20T06:05:05.266132+0000 mon.smithi139 (mon.0) 605 : cluster [INF] 2021-11-20T06:05:05.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:05 smithi139 conmon[33501]: daemon mds.cephfs.smithi139.oudezh restarted 2021-11-20T06:05:05.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:05 smithi139 conmon[33501]: cluster 2021-11-20 2021-11-20T06:05:05.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:05 smithi139 conmon[33501]: T06:05:05.270947+0000 mon.smithi139 (mon.0) 606 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2021-11-20T06:05:05.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:05 smithi178 conmon[36600]: cluster 2021-11-20T06:05:04.232155+0000 mgr.smithi139.dplyxn (mgr.14162) 238 : cluster 2021-11-20T06:05:05.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:05 smithi178 conmon[36600]: [DBG] pgmap v203: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:05:05.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:05 smithi178 conmon[36600]: cluster 2021-11-20T06:05:05.266132+0000 mon.smithi139 (mon.0) 605 : cluster [INF] daemon mds.cephfs.smithi139.oudezh restarted 2021-11-20T06:05:05.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:05 smithi178 conmon[36600]: cluster 2021-11-20T06:05:05.270947+0000 mon.smithi139 (mon. 2021-11-20T06:05:05.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:05 smithi178 conmon[36600]: 0) 606 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2021-11-20T06:05:06.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: cluster 2021-11-20T06:05:05.284751+0000 mon.smithi139 (mon.0) 607 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:05:06.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: 2021-11-20T06:05:06.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: cluster 2021-11-20T06:05:05.287930+0000 mon.smithi139 (mon.0) 608 : cluster [DBG] mds.? [v2:172.21.15.139:6828/957999506,v1:172.21.15.139:6829/957999506] up:boot 2021-11-20T06:05:06.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: cluster 2021-11-20T06:05:05.288000+0000 mon.smithi139 (mon.0) 609 : cluster 2021-11-20T06:05:06.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: [INF] Standby daemon mds.cephfs.smithi139.riczqb assigned to filesystem cephfs as rank 0 2021-11-20T06:05:06.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: cluster 2021-11-20T06:05:05.288137+0000 mon.smithi139 2021-11-20T06:05:06.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: (mon.0) 610 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:05:06.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: cluster 2021-11-20T06:05: 2021-11-20T06:05:06.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: 05.288475+0000 mon.smithi139 (mon.0) 611 : cluster 2021-11-20T06:05:06.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T06:05:06.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: audit 2021-11-20T06:05 2021-11-20T06:05:06.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: :05.288606+0000 mon.smithi139 (mon.0) 612 : audit 2021-11-20T06:05:06.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.oudezh"}]: dispatch 2021-11-20T06:05:06.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: cluster 2021-11-20T06:05:05 2021-11-20T06:05:06.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: .292851+0000 mon.smithi139 (mon.0) 613 : cluster 2021-11-20T06:05:06.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:06 smithi178 conmon[36600]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:replay} 3 up:standby 2021-11-20T06:05:06.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:06 smithi139 conmon[33501]: cluster 2021-11-20T06:05:05.284751+0000 mon.smithi139 (mon.0) 607 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:05:06.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:06 smithi139 conmon[33501]: 2021-11-20T06:05:06.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:06 smithi139 conmon[33501]: cluster 2021-11-20T06:05:05.287930+0000 mon.smithi139 (mon.0) 608 : cluster [DBG] mds.? [v2:172.21.15.139:6828/957999506,v1:172.21.15.139:6829/957999506] up:boot 2021-11-20T06:05:06.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:06 smithi139 conmon[33501]: cluster 2021-11-20T06:05:05.288000+0000 mon.smithi139 (mon.0 2021-11-20T06:05:06.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:06 smithi139 conmon[33501]: ) 609 : cluster [INF] Standby daemon mds.cephfs.smithi139.riczqb assigned to filesystem cephfs as rank 0 2021-11-20T06:05:06.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:06 smithi139 conmon[33501]: cluster 2021-11-20T06:05:05.288137+0000 mon.smithi139 (mon 2021-11-20T06:05:06.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:06 smithi139 conmon[33501]: .0) 610 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:05:06.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:06 smithi139 conmon[33501]: cluster 2021-11-20T06:05:06.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:06 smithi139 conmon[33501]: 2021-11-20T06:05:05.288475+0000 mon.smithi139 (mon.0) 611 : cluster 2021-11-20T06:05:06.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:06 smithi139 conmon[33501]: [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T06:05:06.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:06 smithi139 conmon[33501]: audit 2021-11-20T06:05:05.288606+0000 mon.smithi139 2021-11-20T06:05:06.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:06 smithi139 conmon[33501]: (mon.0) 612 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.oudezh"}]: dispatch 2021-11-20T06:05:06.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:06 smithi139 conmon[33501]: cluster 2021-11-20T06: 2021-11-20T06:05:06.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:06 smithi139 conmon[33501]: 05:05.292851+0000 mon.smithi139 (mon.0) 613 : cluster 2021-11-20T06:05:06.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:06 smithi139 conmon[33501]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:replay} 3 up:standby 2021-11-20T06:05:07.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:07 smithi139 conmon[33501]: cluster 2021-11-20T06:05:06.232582+0000 mgr.smithi139.dplyxn (mgr.14162) 239 2021-11-20T06:05:07.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:07 smithi139 conmon[33501]: : cluster [DBG] pgmap v205: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2021-11-20T06:05:07.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:07 smithi139 conmon[33501]: cluster 2021-11-20T06:05:06.297130+0000 mon.smithi139 (mon.0) 614 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1581019423,v1:172.21.15.139:6827/1581019423] up:reconnect 2021-11-20T06:05:07.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:07 smithi139 conmon[33501]: cluster 2021-11-20T06:05:07.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:07 smithi139 conmon[33501]: 2021-11-20T06:05:06.297201+0000 mon.smithi139 (mon.0) 615 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:reconnect} 3 up:standby 2021-11-20T06:05:07.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:07 smithi178 conmon[36600]: cluster 2021-11-20T06:05:06.232582+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:05:07.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:07 smithi178 conmon[36600]: 14162) 239 : cluster [DBG] pgmap v205: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2021-11-20T06:05:07.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:07 smithi178 conmon[36600]: cluster 2021-11-20T06:05:06.297130+0000 mon.smithi139 (mon.0) 614 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1581019423,v1:172.21.15.139:6827/1581019423] up:reconnect 2021-11-20T06:05:07.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:07 smithi178 conmon[36600]: cluster 2021-11-20T06:05:06.297201+0000 mon.smithi139 (mon.0) 615 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:reconnect} 3 up:standby 2021-11-20T06:05:08.436 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:08 smithi139 conmon[33501]: cluster 2021-11-20T06:05:07.298613+0000 mon.smithi139 (mon.0) 616 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1581019423,v1:172.21.15.139:6827/1581019423] up:rejoin 2021-11-20T06:05:08.436 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:08 smithi139 conmon[33501]: cluster 2021-11-20T06:05:07.298692+0000 mon.smithi139 (mon.0) 617 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:05:08.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:08 smithi178 conmon[36600]: cluster 2021-11-20T06:05:07.298613+0000 mon.smithi139 (mon.0) 616 2021-11-20T06:05:08.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:08 smithi178 conmon[36600]: : cluster [DBG] mds.? [v2:172.21.15.139:6826/1581019423,v1:172.21.15.139:6827/1581019423] up:rejoin 2021-11-20T06:05:08.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:08 smithi178 conmon[36600]: cluster 2021-11-20T06:05:07.298692+0000 mon.smithi139 (mon.0) 617 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:05:09.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:09 smithi178 conmon[36600]: cluster 2021-11-20T06:05:08.233066+0000 mgr.smithi139.dplyxn (mgr.14162) 240 : cluster [DBG] 2021-11-20T06:05:09.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:09 smithi178 conmon[36600]: pgmap v206: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-20T06:05:09.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:09 smithi139 conmon[33501]: cluster 2021-11-20T06:05:08.233066+0000 mgr.smithi139.dplyxn (mgr.14162) 240 : cluster [DBG] 2021-11-20T06:05:09.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:09 smithi139 conmon[33501]: pgmap v206: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-20T06:05:10.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:10 smithi139 conmon[33501]: cluster 2021-11-20T06:05:09.309208+0000 mon.smithi139 (mon.0) 618 : cluster [DBG] mds.? [v2:172.21.15.139:6828/957999506,v1:172.21.15.139:6829/957999506] up:standby 2021-11-20T06:05:10.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:10 smithi139 conmon[33501]: 2021-11-20T06:05:10.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:10 smithi139 conmon[33501]: cluster 2021-11-20T06:05:09.309307+0000 mon.smithi139 (mon.0) 619 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:05:10.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:10 smithi178 conmon[36600]: cluster 2021-11-20T06:05:09.309208+0000 mon.smithi139 (mon.0) 618 : cluster 2021-11-20T06:05:10.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:10 smithi178 conmon[36600]: [DBG] mds.? [v2:172.21.15.139:6828/957999506,v1:172.21.15.139:6829/957999506] up:standby 2021-11-20T06:05:10.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:10 smithi178 conmon[36600]: cluster 2021-11-20T06:05:09.309307+0000 mon.smithi139 (mon.0) 619 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:05:11.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:11 smithi178 conmon[36600]: cluster 2021-11-20T06:05:10.233523+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:05:11.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:11 smithi178 conmon[36600]: 14162) 241 : cluster [DBG] pgmap v207: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:11.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:11 smithi139 conmon[33501]: cluster 2021-11-20T06:05:10.233523+0000 mgr.smithi139.dplyxn (mgr.14162) 241 : cluster [DBG] pgmap v207: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:13.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:13 smithi139 conmon[33501]: cluster 2021-11-20T06:05:12.234048+0000 mgr.smithi139.dplyxn (mgr.14162) 242 : cluster [DBG] pgmap v208: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:13.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:13 smithi178 conmon[36600]: cluster 2021-11-20T06:05:12.234048+0000 mgr.smithi139.dplyxn (mgr.14162) 242 : cluster [DBG] pgmap v208: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:13.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:13 smithi178 conmon[36600]: 2021-11-20T06:05:14.106 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:14.121+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:14.107 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:14.122+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T06:05:14.108 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:14.122+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T06:05:14.108 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:14.122+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T06:05:14.109 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:14.122+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T06:05:15.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:15 smithi139 conmon[33501]: cluster 2021-11-20T06:05:14.234516+0000 mgr.smithi139.dplyxn (mgr.14162) 243 : cluster [DBG] pgmap v209: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:15.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:15 smithi178 conmon[36600]: cluster 2021-11-20T06:05:14.234516+0000 mgr.smithi139.dplyxn (mgr.14162) 243 : cluster 2021-11-20T06:05:15.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:15 smithi178 conmon[36600]: [DBG] pgmap v209: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:17.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:17 smithi139 conmon[33501]: cluster 2021-11-20T06:05:16.235010+0000 mgr.smithi139.dplyxn (mgr.14162) 244 : cluster [DBG] pgmap v210: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:05:17.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:17 smithi178 conmon[36600]: cluster 2021-11-20T06:05:16.235010+0000 mgr.smithi139.dplyxn (mgr.14162) 244 : cluster [DBG] pgmap v210: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:05:17.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:17 smithi178 conmon[36600]: 2021-11-20T06:05:19.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:19 smithi178 conmon[36600]: cluster 2021-11-20T06:05:18.235378+0000 mgr.smithi139.dplyxn (mgr.14162) 245 : cluster [DBG] 2021-11-20T06:05:19.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:19 smithi178 conmon[36600]: pgmap v211: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T06:05:19.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:19 smithi139 conmon[33501]: cluster 2021-11-20T06:05:18.235378+0000 mgr.smithi139.dplyxn (mgr.14162) 245 : 2021-11-20T06:05:19.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:19 smithi139 conmon[33501]: cluster [DBG] pgmap v211: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T06:05:21.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:21 smithi178 conmon[36600]: cluster 2021-11-20T06:05:20.235851+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:05:21.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:21 smithi178 conmon[36600]: ) 246 : cluster [DBG] pgmap v212: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2021-11-20T06:05:21.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:21 smithi139 conmon[33501]: cluster 2021-11-20T06:05:20.235851+0000 mgr.smithi139.dplyxn (mgr.14162) 246 : cluster [DBG] pgmap v212: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2021-11-20T06:05:22.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:22 smithi178 conmon[36600]: cluster 2021-11-20T06:05:22.201122+0000 mon.smithi139 (mon.0) 620 : cluster [WRN] Replacing daemon mds.cephfs.smithi139.riczqb as rank 0 with standby daemon mds.cephfs.smithi139.oudezh 2021-11-20T06:05:22.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:22 smithi178 conmon[36600]: cluster 2021-11-20T06:05:22.201205+0000 mon.smithi139 (mon.0) 621 : cluster [INF] MDS daemon mds.cephfs.smithi139.riczqb is removed because it is dead or otherwise unavailable. 2021-11-20T06:05:22.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:22 smithi178 conmon[36600]: cluster 2021-11-20T06:05:22.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:22 smithi178 conmon[36600]: 2021-11-20T06:05:22.208480+0000 mon.smithi139 (mon.0) 622 2021-11-20T06:05:22.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:22 smithi178 conmon[36600]: : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2021-11-20T06:05:22.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:22 smithi178 conmon[36600]: cluster 2021-11-20T06:05:22.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:22 smithi178 conmon[36600]: 2021-11-20T06: 2021-11-20T06:05:22.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:22 smithi178 conmon[36600]: 05:22.213658+0000 mon.smithi139 (mon.0) 2021-11-20T06:05:22.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:22 smithi178 conmon[36600]: 623 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:replay} 2 up:standby 2021-11-20T06:05:22.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:22 smithi139 conmon[33501]: cluster 2021-11-20T06:05:22.201122+0000 mon.smithi139 (mon.0) 620 : cluster [WRN] Replacing daemon mds.cephfs.smithi139.riczqb as rank 0 with standby daemon mds.cephfs.smithi139.oudezh 2021-11-20T06:05:22.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:22 smithi139 conmon[33501]: cluster 2021-11-20T06:05:22.201205+0000 mon.smithi139 (mon.0) 621 : cluster [INF] MDS daemon mds.cephfs.smithi139.riczqb is removed because it is dead or otherwise unavailable. 2021-11-20T06:05:22.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:22 smithi139 conmon[33501]: cluster 2021-11-20T06:05:22.208480+0000 mon.smithi139 (mon.0) 622 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2021-11-20T06:05:22.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:22 smithi139 conmon[33501]: cluster 2021-11-20T06:05:22.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:22 smithi139 conmon[33501]: 2021-11-20T06:05:22.213658+0000 mon.smithi139 (mon.0) 623 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:replay} 2 up:standby 2021-11-20T06:05:23.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:23 smithi139 conmon[33501]: cluster 2021-11-20T06:05:22.236459+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:05:23.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:23 smithi139 conmon[33501]: ) 247 : cluster [DBG] pgmap v214: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:05:23.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:23 smithi139 conmon[33501]: cluster 2021-11-20T06:05:23.218026+0000 mon.smithi139 (mon.0) 624 : cluster [DBG] mds.? [v2:172.21.15.139:6828/957999506,v1:172.21.15.139:6829/957999506] up:reconnect 2021-11-20T06:05:23.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:23 smithi139 conmon[33501]: cluster 2021-11-20T06:05:23.218088+0000 mon.smithi139 (mon.0) 625 : cluster [DBG] 2021-11-20T06:05:23.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:23 smithi139 conmon[33501]: mds.? [v2:172.21.15.139:6826/1132764783,v1:172.21.15.139:6827/1132764783] up:boot 2021-11-20T06:05:23.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:23 smithi139 conmon[33501]: cluster 2021- 2021-11-20T06:05:23.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:23 smithi139 conmon[33501]: 11-20T06:05:23.218172+0000 mon.smithi139 (mon.0) 626 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:reconnect} 3 up:standby 2021-11-20T06:05:23.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:23 smithi139 conmon[33501]: audit 2021-11-20 2021-11-20T06:05:23.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:23 smithi139 conmon[33501]: T06:05:23.218321+0000 mon.smithi139 (mon.0) 627 : audit [DBG] 2021-11-20T06:05:23.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:23 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.riczqb"}]: dispatch 2021-11-20T06:05:23.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:23 smithi178 conmon[36600]: cluster 2021-11-20T06:05:22.236459+0000 mgr.smithi139.dplyxn (mgr.14162) 247 : cluster [DBG] 2021-11-20T06:05:23.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:23 smithi178 conmon[36600]: pgmap v214: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:05:23.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:23 smithi178 conmon[36600]: cluster 2021-11-20T06:05:23.218026+0000 mon.smithi139 (mon.0) 624 : cluster [DBG] mds.? [v2:172.21.15.139:6828/957999506,v1:172.21.15.139:6829/957999506] up:reconnect 2021-11-20T06:05:23.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:23 smithi178 conmon[36600]: cluster 2021-11-20T06 2021-11-20T06:05:23.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:23 smithi178 conmon[36600]: :05:23.218088+0000 mon.smithi139 (mon.0) 625 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1132764783,v1:172.21.15.139:6827/1132764783] up:boot 2021-11-20T06:05:23.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:23 smithi178 conmon[36600]: cluster 2021-11-20T 2021-11-20T06:05:23.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:23 smithi178 conmon[36600]: 06:05:23.218172+0000 mon.smithi139 (mon.0) 626 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:reconnect} 3 up:standby 2021-11-20T06:05:23.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:23 smithi178 conmon[36600]: audit 2021 2021-11-20T06:05:23.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:23 smithi178 conmon[36600]: -11-20T06:05:23. 2021-11-20T06:05:23.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:23 smithi178 conmon[36600]: 218321+0000 mon.smithi139 (mon.0) 627 2021-11-20T06:05:23.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:23 smithi178 conmon[36600]: : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.riczqb"}]: dispatch 2021-11-20T06:05:25.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:25 smithi139 conmon[33501]: cluster 2021-11-20T06:05:24.219313+0000 mon.smithi139 (mon.0) 628 : cluster 2021-11-20T06:05:25.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:25 smithi139 conmon[33501]: [DBG] mds.? [v2:172.21.15.139:6828/957999506,v1:172.21.15.139:6829/957999506] up:rejoin 2021-11-20T06:05:25.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:25 smithi139 conmon[33501]: cluster 2021-11-20T06:05:24.219386+0000 mon.smithi139 (mon.0) 629 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:05:25.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:25 smithi139 conmon[33501]: cluster 2021-11-20T06:05:24.237092+0000 2021-11-20T06:05:25.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:25 smithi139 conmon[33501]: mgr.smithi139.dplyxn (mgr.14162) 248 : cluster [DBG] pgmap v215: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T06:05:25.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:25 smithi178 conmon[36600]: cluster 2021-11-20T06:05:24.219313+0000 mon.smithi139 (mon.0) 628 : cluster [DBG] mds.? [v2:172.21.15.139:6828/957999506,v1:172.21.15.139:6829/957999506] up:rejoin 2021-11-20T06:05:25.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:25 smithi178 conmon[36600]: cluster 2021 2021-11-20T06:05:25.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:25 smithi178 conmon[36600]: -11-20T06:05:24.219386+0000 mon.smithi139 (mon.0) 629 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:05:25.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:25 smithi178 conmon[36600]: cluster 2021-11-20T06:05:24.237092+0000 mgr.smithi139.dplyxn (mgr.14162) 248 : cluster [DBG] pgmap v215: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T06:05:27.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:27 smithi139 conmon[33501]: cluster 2021-11-20T06:05:26.237729+0000 mgr.smithi139.dplyxn (mgr.14162) 249 : cluster 2021-11-20T06:05:27.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:27 smithi139 conmon[33501]: [DBG] pgmap v216: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2021-11-20T06:05:27.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:27 smithi178 conmon[36600]: cluster 2021-11-20T06:05:26.237729+0000 mgr.smithi139.dplyxn (mgr.14162) 249 : cluster [DBG] pgmap v216: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2021-11-20T06:05:28.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:28 smithi139 conmon[33501]: cluster 2021-11-20T06:05:27.296185+0000 mon.smithi139 (mon.0) 630 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1132764783,v1:172.21.15.139:6827/1132764783] up:standby 2021-11-20T06:05:28.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:28 smithi139 conmon[33501]: cluster 2021-11-20T06:05:27.296233+0000 mon.smithi139 (mon.0) 631 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:05:28.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:28 smithi178 conmon[36600]: cluster 2021-11-20T06:05:27.296185+0000 mon.smithi139 (mon.0) 630 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1132764783,v1:172.21.15.139:6827/1132764783] up:standby 2021-11-20T06:05:28.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:28 smithi178 conmon[36600]: cluster 2021-11-20T06:05:27.296233+0000 mon.smithi139 (mon.0) 631 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:05:29.108 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:29.123+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:29.109 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:29.124+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T06:05:29.109 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:29.124+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T06:05:29.109 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:29.124+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T06:05:29.110 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:29.124+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T06:05:29.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:29 smithi178 conmon[36600]: cluster 2021-11-20T06:05:28.238374+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:05:29.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:29 smithi178 conmon[36600]: 250 : cluster [DBG] pgmap v217: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:29.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:29 smithi139 conmon[33501]: cluster 2021-11-20T06:05:28.238374+0000 mgr.smithi139.dplyxn (mgr.14162) 250 : cluster [DBG] pgmap v217: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:31.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:31 smithi178 conmon[36600]: cluster 2021-11-20T06:05:30.239044+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:05:31.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:31 smithi178 conmon[36600]: 251 : cluster [DBG] pgmap v218: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:31.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:31 smithi178 conmon[36600]: audit 2021-11-20T06:05:31.255153+0000 mon.smithi139 (mon.0) 632 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:05:31.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:31 smithi178 conmon[36600]: audit 2021-11-20T06: 2021-11-20T06:05:31.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:31 smithi178 conmon[36600]: 05:31.255505+0000 mon.smithi139 (mon.0) 633 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:05:31.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:31 smithi139 conmon[33501]: cluster 2021-11-20T06:05:30.239044+0000 mgr.smithi139.dplyxn (mgr.14162) 251 : cluster [DBG] pgmap v218: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:31.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:31 smithi139 conmon[33501]: audit 2021-11-20T06:05:31.255153+0000 mon.smithi139 (mon.0) 632 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:05:31.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:31 smithi139 conmon[33501]: audit 2021-11-20T06:05:31.255505+0000 mon.smithi139 (mon.0) 633 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:05:33.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:33 smithi139 conmon[33501]: cluster 2021-11-20T06:05:32.239553+0000 mgr.smithi139.dplyxn (mgr.14162) 252 : cluster [DBG] pgmap v219: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:33.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:33 smithi178 conmon[36600]: cluster 2021-11-20T06:05:32.239553+0000 mgr.smithi139.dplyxn (mgr.14162) 252 : cluster [DBG] pgmap v219: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:34.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:34 smithi139 conmon[33501]: debug 2021-11-20T06:05:34.326+0000 7f2a21654700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:05:34.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:34 smithi139 conmon[33501]: cluster 2021-11-20T06:05:34. 2021-11-20T06:05:34.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:34 smithi139 conmon[33501]: 277656+0000 mon.smithi139 (mon.0) 634 : cluster [INF] daemon mds.cephfs.smithi139.oudezh restarted 2021-11-20T06:05:34.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:34 smithi139 conmon[33501]: cluster 2021-11-20T06:05:34.281485+0000 2021-11-20T06:05:34.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:34 smithi139 conmon[33501]: mon.smithi139 (mon.0) 635 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2021-11-20T06:05:34.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:34 smithi178 conmon[36600]: cluster 2021-11-20T06:05:34.277656+0000 mon.smithi139 (mon.0) 634 : cluster [INF] daemon mds.cephfs.smithi139.oudezh restarted 2021-11-20T06:05:34.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:34 smithi178 conmon[36600]: cluster 2021-11-20T06:05:34.281485+0000 mon.smithi139 (mon.0) 635 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2021-11-20T06:05:35.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: cluster 2021-11-20T06:05:34.239956+0000 mgr.smithi139.dplyxn (mgr.14162) 253 : cluster [DBG] pgmap v220: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:05:35.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: 2021-11-20T06:05:35.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: cluster 2021-11-20T06:05:34.328363+0000 mon.smithi139 (mon.0) 636 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:05:35.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: cluster 2021 2021-11-20T06:05:35.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: -11-20T06:05:34.331626+0000 mon.smithi139 (mon.0) 637 : cluster [DBG] 2021-11-20T06:05:35.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: mds.? [v2:172.21.15.139:6828/1072553578,v1:172.21.15.139:6829/1072553578] up:boot 2021-11-20T06:05:35.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: cluster 2021-11 2021-11-20T06:05:35.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: -20T06:05:34.331695+0000 mon.smithi139 2021-11-20T06:05:35.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: (mon.0) 638 : cluster [INF] 2021-11-20T06:05:35.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: Standby daemon mds.cephfs.smithi139.riczqb assigned to filesystem cephfs as rank 0 2021-11-20T06:05:35.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: cluster 2021- 2021-11-20T06:05:35.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: 11-20T06:05:34.331792+0000 2021-11-20T06:05:35.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: mon.smithi139 (mon.0) 639 : cluster 2021-11-20T06:05:35.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:05:35.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: cluster 2021- 2021-11-20T06:05:35.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: 11-20T06:05:34.332051+0000 2021-11-20T06:05:35.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: mon.smithi139 (mon.0) 640 : cluster 2021-11-20T06:05:35.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T06:05:35.697 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: audit 2021-11- 2021-11-20T06:05:35.697 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: 20T06:05:34.332157+0000 mon.smithi139 2021-11-20T06:05:35.697 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: (mon.0) 641 : audit [DBG] 2021-11-20T06:05:35.697 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.oudezh"}]: dispatch 2021-11-20T06:05:35.698 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: cluster 2021-11 2021-11-20T06:05:35.698 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: -20T06:05:34.335223+0000 2021-11-20T06:05:35.698 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: mon.smithi139 (mon.0) 642 : cluster [DBG] 2021-11-20T06:05:35.698 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:35 smithi178 conmon[36600]: fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:replay} 3 up:standby 2021-11-20T06:05:35.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:35 smithi139 conmon[33501]: cluster 2021-11-20T06:05:34.239956+0000 mgr.smithi139.dplyxn (mgr.14162) 253 : cluster [DBG] pgmap v220: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:05:35.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:35 smithi139 conmon[33501]: 2021-11-20T06:05:35.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:35 smithi139 conmon[33501]: cluster 2021-11-20T06:05:34.328363+0000 mon.smithi139 (mon.0) 636 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:05:35.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:35 smithi139 conmon[33501]: cluster 2021-11-20T06:05:34.331626+0000 mon.smithi139 (mon.0) 637 : cluster [DBG] mds.? [v2:172.21.15.139:6828/1072553578,v1:172.21.15.139:6829/1072553578] up:boot 2021-11-20T06:05:35.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:35 smithi139 conmon[33501]: cluster 2021-11-20T06:05:34.331695+0000 mon.smithi139 (mon.0) 638 : cluster [INF] Standby daemon mds.cephfs.smithi139.riczqb assigned to filesystem cephfs as rank 0 2021-11-20T06:05:35.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:35 smithi139 conmon[33501]: cluster 2021-11-20T06:05:34.331792+0000 mon.smithi139 (mon.0) 2021-11-20T06:05:35.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:35 smithi139 conmon[33501]: 639 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:05:35.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:35 smithi139 conmon[33501]: cluster 2021-11-20T06:05 2021-11-20T06:05:35.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:35 smithi139 conmon[33501]: :34.332051+0000 mon.smithi139 (mon.0) 640 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T06:05:35.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:35 smithi139 conmon[33501]: 2021-11-20T06:05:35.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:35 smithi139 conmon[33501]: audit 2021-11-20T06:05:34.332157+0000 2021-11-20T06:05:35.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:35 smithi139 conmon[33501]: mon.smithi139 (mon.0) 641 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.oudezh"}]: dispatch 2021-11-20T06:05:35.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:35 smithi139 conmon[33501]: cluster 2021-11-20T06:05:34.335223+0000 mon.smithi139 (mon.0 2021-11-20T06:05:35.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:35 smithi139 conmon[33501]: ) 642 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:replay} 3 up:standby 2021-11-20T06:05:36.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:36 smithi139 conmon[33501]: cluster 2021-11-20T06:05:35.341050+0000 mon.smithi139 (mon.0) 2021-11-20T06:05:36.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:36 smithi139 conmon[33501]: 643 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1132764783,v1:172.21.15.139:6827/1132764783] up:reconnect 2021-11-20T06:05:36.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:36 smithi139 conmon[33501]: cluster 2021-11-20T06:05:35.341120+0000 mon.smithi139 (mon.0) 644 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:reconnect} 3 up:standby 2021-11-20T06:05:36.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:36 smithi178 conmon[36600]: cluster 2021-11-20T06:05:35.341050+0000 mon.smithi139 (mon.0) 643 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1132764783,v1:172.21.15.139:6827/1132764783] up:reconnect 2021-11-20T06:05:36.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:36 smithi178 conmon[36600]: cluster 2021-11-20T06:05:35.341120+0000 mon.smithi139 (mon.0) 644 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:reconnect} 3 up:standby 2021-11-20T06:05:37.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:37 smithi139 conmon[33501]: cluster 2021-11-20T06:05:36.243825+0000 mgr.smithi139.dplyxn (mgr.14162) 254 2021-11-20T06:05:37.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:37 smithi139 conmon[33501]: : cluster [DBG] pgmap v222: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:05:37.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:37 smithi139 conmon[33501]: cluster 2021-11-20T06:05:36.345314+0000 mon.smithi139 (mon.0) 645 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1132764783,v1:172.21.15.139:6827/1132764783] up:rejoin 2021-11-20T06:05:37.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:37 smithi139 conmon[33501]: cluster 2021-11-20T06:05:36.345362+0000 mon.smithi139 (mon.0) 646 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:05:37.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:37 smithi178 conmon[36600]: cluster 2021-11-20T06:05:36.243825+0000 mgr.smithi139.dplyxn (mgr.14162) 254 : cluster 2021-11-20T06:05:37.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:37 smithi178 conmon[36600]: [DBG] pgmap v222: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:05:37.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:37 smithi178 conmon[36600]: cluster 2021-11-20T06:05:36.345314+0000 mon.smithi139 (mon.0) 645 : cluster [DBG] mds.? [v2:172.21.15.139:6826/1132764783,v1:172.21.15.139:6827/1132764783] up:rejoin 2021-11-20T06:05:37.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:37 smithi178 conmon[36600]: cluster 2021-11-20T06:05: 2021-11-20T06:05:37.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:37 smithi178 conmon[36600]: 36.345362+0000 mon.smithi139 (mon.0) 2021-11-20T06:05:37.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:37 smithi178 conmon[36600]: 646 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:05:39.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:39 smithi178 conmon[36600]: cluster 2021-11-20T06:05:38.245157+0000 mgr.smithi139.dplyxn (mgr.14162) 255 : cluster 2021-11-20T06:05:39.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:39 smithi178 conmon[36600]: [DBG] pgmap v223: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:05:39.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:39 smithi178 conmon[36600]: cluster 2021-11-20T06:05:38.353890+0000 mon.smithi139 (mon.0) 647 : cluster [DBG] mds.? [v2:172.21.15.139:6828/1072553578,v1:172.21.15.139:6829/1072553578] up:standby 2021-11-20T06:05:39.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:39 smithi178 conmon[36600]: cluster 2021-11-20T06:05: 2021-11-20T06:05:39.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:39 smithi178 conmon[36600]: 38.353949+0000 mon.smithi139 (mon.0) 648 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:05:39.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:39 smithi139 conmon[33501]: cluster 2021-11-20T06:05:38.245157+0000 mgr.smithi139.dplyxn (mgr.14162) 255 : cluster 2021-11-20T06:05:39.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:39 smithi139 conmon[33501]: [DBG] pgmap v223: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:05:39.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:39 smithi139 conmon[33501]: cluster 2021-11-20T06:05:38.353890+0000 mon.smithi139 (mon.0) 647 : cluster [DBG] mds.? [v2:172.21.15.139:6828/1072553578,v1:172.21.15.139:6829/1072553578] up:standby 2021-11-20T06:05:39.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:39 smithi139 conmon[33501]: cluster 2021-11-20T06:05:38.353949+0000 mon.smithi139 (mon.0) 648 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:05:41.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:41 smithi178 conmon[36600]: cluster 2021-11-20T06:05:40.245626+0000 mgr.smithi139.dplyxn (mgr.14162) 256 : cluster [DBG] pgmap v224: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:41.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:41 smithi139 conmon[33501]: cluster 2021-11-20T06:05:40.245626+0000 mgr.smithi139.dplyxn (mgr.14162) 256 : cluster [DBG] 2021-11-20T06:05:41.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:41 smithi139 conmon[33501]: pgmap v224: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:43.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:43 smithi139 conmon[33501]: cluster 2021-11-20T06:05:42.246204+0000 mgr.smithi139.dplyxn (mgr.14162) 257 : cluster [DBG] pgmap v225: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:43.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:43 smithi178 conmon[36600]: cluster 2021-11-20T06:05:42.246204+0000 mgr.smithi139.dplyxn (mgr.14162) 257 : cluster [DBG] pgmap v225: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:44.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:44.126+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:44.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:44.126+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-20T06:05:44.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:44.126+0000 7f2596ffd700 1 -- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 msgr2=0x55cd16db8170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-20T06:05:44.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:44.126+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-20T06:05:44.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:44.126+0000 7f2596ffd700 1 --2- 172.21.15.139:0/3754453719 >> [v2:172.21.15.178:6824/1867304351,v1:172.21.15.178:6825/1867304351] conn(0x55cd16db5d80 0x55cd16db8170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-20T06:05:45.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:45 smithi178 conmon[36600]: cluster 2021-11-20T06:05:44.246722+0000 mgr.smithi139.dplyxn (mgr.14162) 258 : cluster [DBG] pgmap v226: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:45.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:45 smithi139 conmon[33501]: cluster 2021-11-20T06:05:44.246722+0000 mgr.smithi139.dplyxn (mgr.14162) 258 : cluster [DBG] pgmap v226: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:47.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:47 smithi178 conmon[36600]: cluster 2021-11-20T06:05:46.247206+0000 mgr.smithi139.dplyxn (mgr.14162) 259 2021-11-20T06:05:47.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:47 smithi178 conmon[36600]: : cluster [DBG] pgmap v227: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:05:47.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:47 smithi178 conmon[36600]: audit 2021-11-20T06:05:46.772207+0000 mon.smithi139 (mon.0) 649 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:05:47.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:47 smithi178 conmon[36600]: audit 2021-11 2021-11-20T06:05:47.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:47 smithi178 conmon[36600]: -20T06:05:47.180710+0000 mon.smithi139 (mon.0) 650 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:05:47.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:47 smithi139 conmon[33501]: cluster 2021-11-20T06:05:46.247206+0000 mgr.smithi139.dplyxn (mgr.14162) 259 : cluster [DBG] pgmap v227: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:05:47.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:47 smithi139 conmon[33501]: audit 2021-11 2021-11-20T06:05:47.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:47 smithi139 conmon[33501]: -20T06:05:46.772207+0000 mon.smithi139 (mon.0) 649 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:05:47.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:47 smithi139 conmon[33501]: audit 2021-11-20T06:05:47.180710+0000 mon.smithi139 (mon.0) 650 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:05:48.584 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:48 smithi139 conmon[33501]: debug 2021-11-20T06:05:48.303+0000 7f2a21654700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:05:48.585 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:48 smithi139 conmon[33501]: cluster 2021-11-20T06:05: 2021-11-20T06:05:48.585 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:48 smithi139 conmon[33501]: 47.342110+0000 mon.smithi139 (mon 2021-11-20T06:05:48.586 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:48 smithi139 conmon[33501]: .0) 651 : cluster [INF] daemon mds.cephfs.smithi139.riczqb restarted 2021-11-20T06:05:48.586 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:48 smithi139 conmon[33501]: cluster 2021-11- 2021-11-20T06:05:48.586 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:48 smithi139 conmon[33501]: 20T06:05:47.345994+0000 mon.smithi139 (mon.0 2021-11-20T06:05:48.586 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:48 smithi139 conmon[33501]: ) 652 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2021-11-20T06:05:48.587 DEBUG:teuthology.orchestra.run:got remote process result: 124 2021-11-20T06:05:48.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:48 smithi178 conmon[36600]: cluster 2021-11-20T06:05:47.342110+0000 mon.smithi139 (mon.0) 651 : cluster [INF] daemon mds.cephfs.smithi139.riczqb restarted 2021-11-20T06:05:48.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:48 smithi178 conmon[36600]: 2021-11-20T06:05:48.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:48 smithi178 conmon[36600]: cluster 2021-11-20T06:05:47.345994+0000 mon.smithi139 (mon.0) 652 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2021-11-20T06:05:49.590 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:05:49.621 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:49 smithi139 conmon[33501]: cluster 2021-11-20T06:05:48.247840+0000 mgr.smithi139.dplyxn (mgr.14162) 260 : cluster [DBG] pgmap v229: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T06:05:49.622 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:49 smithi139 conmon[33501]: cluster 2021-11-20T06:05:48.304236+0000 2021-11-20T06:05:49.622 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:49 smithi139 conmon[33501]: mon.smithi139 (mon.0) 653 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:05:49.622 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:49 smithi139 conmon[33501]: cluster 2021-11-20T06:05:48.310491+0000 mon.smithi139 (mon.0) 654 : cluster [DBG] mds.? [v2:172.21.15.139:6826/3938771458,v1:172.21.15.139:6827/3938771458] up:boot 2021-11-20T06:05:49.623 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:49 smithi139 conmon[33501]: cluster 2021-11-20T06:05:48.310549+0000 mon.smithi139 (mon.0) 655 : cluster [INF] Standby daemon mds.cephfs.smithi139.oudezh assigned to filesystem cephfs as rank 0 2021-11-20T06:05:49.623 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:49 smithi139 conmon[33501]: cluster 2021-11-20T06:05:48.310672+0000 mon.smithi139 (mon.0) 656 : 2021-11-20T06:05:49.623 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:49 smithi139 conmon[33501]: cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:05:49.623 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:49 smithi139 conmon[33501]: cluster 2021-11-20T06:05 2021-11-20T06:05:49.624 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:49 smithi139 conmon[33501]: :48.311281+0000 mon.smithi139 (mon.0) 657 : cluster 2021-11-20T06:05:49.624 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:49 smithi139 conmon[33501]: [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T06:05:49.624 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:49 smithi139 conmon[33501]: audit 2021-11-20T06:05: 2021-11-20T06:05:49.624 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:49 smithi139 conmon[33501]: 48.311389+0000 mon.smithi139 (mon.0) 658 : audit [DBG] 2021-11-20T06:05:49.625 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:49 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.riczqb"}]: dispatch 2021-11-20T06:05:49.625 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:49 smithi139 conmon[33501]: cluster 2021-11-20T06:05:48.316111+0000 mon.smithi139 ( 2021-11-20T06:05:49.625 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:49 smithi139 conmon[33501]: mon.0) 659 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:replay} 3 up:standby 2021-11-20T06:05:49.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: cluster 2021-11-20T06:05:48.247840+0000 mgr.smithi139.dplyxn (mgr.14162) 260 : cluster [DBG] pgmap v229: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T06:05:49.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: cluster 2021-11-20T06:05:48.304236+0000 mon.smithi139 (mon.0) 653 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:05:49.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: cluster 2021-11-20T06:05:48.310491+0000 2021-11-20T06:05:49.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: mon.smithi139 (mon.0) 654 : cluster [DBG] mds.? [v2:172.21.15.139:6826/3938771458,v1:172.21.15.139:6827/3938771458] up:boot 2021-11-20T06:05:49.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: cluster 2021-11 2021-11-20T06:05:49.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: -20T06:05:48.310549+0000 2021-11-20T06:05:49.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: mon.smithi139 (mon.0) 655 : cluster 2021-11-20T06:05:49.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: [INF] Standby daemon mds.cephfs.smithi139.oudezh assigned to filesystem cephfs as rank 0 2021-11-20T06:05:49.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: cluster 2021-11-20 2021-11-20T06:05:49.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: T06:05:48.310672+0000 mon.smithi139 ( 2021-11-20T06:05:49.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: mon.0) 656 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:05:49.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: 2021-11-20T06:05:49.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: cluster 2021-11-20T06:05:49.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: 2021-11-20T06:05: 2021-11-20T06:05:49.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: 48.311281+0000 mon.smithi139 (mon.0 2021-11-20T06:05:49.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: ) 657 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T06:05:49.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: 2021-11-20T06:05:49.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: audit 2021-11-20T06:05:49.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: 2021-11-20T06:05: 2021-11-20T06:05:49.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: 48.311389+0000 mon.smithi139 (mon.0) 658 : audit 2021-11-20T06:05:49.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.riczqb"}]: dispatch 2021-11-20T06:05:49.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: cluster 2021-11-20T06:05:48.316111+0000 2021-11-20T06:05:49.697 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: mon.smithi139 (mon.0) 659 : cluster [DBG] 2021-11-20T06:05:49.697 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:49 smithi178 conmon[36600]: fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:replay} 3 up:standby 2021-11-20T06:05:49.723 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.737+0000 7efc4fc0c700 1 -- 172.21.15.139:0/2789563699 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc480f4960 msgr2=0x7efc480f4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:49.724 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.737+0000 7efc4fc0c700 1 --2- 172.21.15.139:0/2789563699 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc480f4960 0x7efc480f4d80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7efc44004660 tx=0x7efc44009b30).stop 2021-11-20T06:05:49.724 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.737+0000 7efc4fc0c700 1 -- 172.21.15.139:0/2789563699 shutdown_connections 2021-11-20T06:05:49.724 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.737+0000 7efc4fc0c700 1 --2- 172.21.15.139:0/2789563699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc480f5300 0x7efc480f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:05:49.724 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.737+0000 7efc4fc0c700 1 --2- 172.21.15.139:0/2789563699 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc480f4960 0x7efc480f4d80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:49.725 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.737+0000 7efc4fc0c700 1 -- 172.21.15.139:0/2789563699 >> 172.21.15.139:0/2789563699 conn(0x7efc480f0440 msgr2=0x7efc480f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:49.725 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.737+0000 7efc4fc0c700 1 -- 172.21.15.139:0/2789563699 shutdown_connections 2021-11-20T06:05:49.725 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.737+0000 7efc4fc0c700 1 -- 172.21.15.139:0/2789563699 wait complete. 2021-11-20T06:05:49.725 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.737+0000 7efc4fc0c700 1 Processor -- start 2021-11-20T06:05:49.726 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.737+0000 7efc4fc0c700 1 -- start start 2021-11-20T06:05:49.726 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.737+0000 7efc4fc0c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc480f4960 0x7efc48106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:49.726 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.737+0000 7efc4fc0c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc480f5300 0x7efc481073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:49.726 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.737+0000 7efc4fc0c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc48107a20 con 0x7efc480f5300 2021-11-20T06:05:49.727 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.737+0000 7efc4fc0c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc48107b60 con 0x7efc480f4960 2021-11-20T06:05:49.727 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.738+0000 7efc4d1a7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc480f5300 0x7efc481073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:49.727 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.738+0000 7efc4d1a7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc480f5300 0x7efc481073e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57934/0 (socket says 172.21.15.139:57934) 2021-11-20T06:05:49.727 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.738+0000 7efc4d1a7700 1 -- 172.21.15.139:0/2694673531 learned_addr learned my addr 172.21.15.139:0/2694673531 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:05:49.728 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.738+0000 7efc4d9a8700 1 --2- 172.21.15.139:0/2694673531 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc480f4960 0x7efc48106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:49.728 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.738+0000 7efc4d1a7700 1 -- 172.21.15.139:0/2694673531 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc480f4960 msgr2=0x7efc48106ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:49.728 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.738+0000 7efc4d1a7700 1 --2- 172.21.15.139:0/2694673531 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc480f4960 0x7efc48106ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:49.728 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.738+0000 7efc4d1a7700 1 -- 172.21.15.139:0/2694673531 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc44005040 con 0x7efc480f5300 2021-11-20T06:05:49.728 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.738+0000 7efc4d9a8700 1 --2- 172.21.15.139:0/2694673531 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc480f4960 0x7efc48106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:05:49.729 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.738+0000 7efc4d1a7700 1 --2- 172.21.15.139:0/2694673531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc480f5300 0x7efc481073e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7efc380093f0 tx=0x7efc38006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:49.729 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.738+0000 7efc3e7fc700 1 -- 172.21.15.139:0/2694673531 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc38013640 con 0x7efc480f5300 2021-11-20T06:05:49.729 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.738+0000 7efc3e7fc700 1 -- 172.21.15.139:0/2694673531 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc38009e40 con 0x7efc480f5300 2021-11-20T06:05:49.729 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.738+0000 7efc4fc0c700 1 -- 172.21.15.139:0/2694673531 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc4810c600 con 0x7efc480f5300 2021-11-20T06:05:49.730 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.738+0000 7efc3e7fc700 1 -- 172.21.15.139:0/2694673531 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc3800dbf0 con 0x7efc480f5300 2021-11-20T06:05:49.730 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.738+0000 7efc4fc0c700 1 -- 172.21.15.139:0/2694673531 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc4810cbc0 con 0x7efc480f5300 2021-11-20T06:05:49.731 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.739+0000 7efc4fc0c700 1 -- 172.21.15.139:0/2694673531 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc48027a00 con 0x7efc480f5300 2021-11-20T06:05:49.731 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.739+0000 7efc3e7fc700 1 -- 172.21.15.139:0/2694673531 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7efc3801a060 con 0x7efc480f5300 2021-11-20T06:05:49.732 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.740+0000 7efc3e7fc700 1 --2- 172.21.15.139:0/2694673531 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7efc3405c2a0 0x7efc3405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:49.732 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.740+0000 7efc3e7fc700 1 -- 172.21.15.139:0/2694673531 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7efc38076100 con 0x7efc480f5300 2021-11-20T06:05:49.732 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.740+0000 7efc4d9a8700 1 --2- 172.21.15.139:0/2694673531 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7efc3405c2a0 0x7efc3405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:49.732 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.740+0000 7efc4d9a8700 1 --2- 172.21.15.139:0/2694673531 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7efc3405c2a0 0x7efc3405e760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7efc44009d60 tx=0x7efc44009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:49.733 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.744+0000 7efc3e7fc700 1 -- 172.21.15.139:0/2694673531 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc38048b90 con 0x7efc480f5300 2021-11-20T06:05:49.913 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.928+0000 7efc4fc0c700 1 -- 172.21.15.139:0/2694673531 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc480feef0 con 0x7efc480f5300 2021-11-20T06:05:49.914 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.929+0000 7efc3e7fc700 1 -- 172.21.15.139:0/2694673531 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+2947 (secure 0 0 0) 0x7efc3804c1b0 con 0x7efc480f5300 2021-11-20T06:05:49.914 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:05:49.915 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":48},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":50,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:05:49.314169+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14560":{"gid":14560,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":49,"state":"up:reconnect","state_seq":5,"addr":"172.21.15.139:6829/1072553578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1072553578},{"type":"v1","addr":"172.21.15.139:6829","nonce":1072553578}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:05:49.916 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.931+0000 7efc4fc0c700 1 -- 172.21.15.139:0/2694673531 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7efc3405c2a0 msgr2=0x7efc3405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:49.916 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.931+0000 7efc4fc0c700 1 --2- 172.21.15.139:0/2694673531 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7efc3405c2a0 0x7efc3405e760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7efc44009d60 tx=0x7efc44009b30).stop 2021-11-20T06:05:49.916 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.931+0000 7efc4fc0c700 1 -- 172.21.15.139:0/2694673531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc480f5300 msgr2=0x7efc481073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:49.917 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.931+0000 7efc4fc0c700 1 --2- 172.21.15.139:0/2694673531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc480f5300 0x7efc481073e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7efc380093f0 tx=0x7efc38006ec0).stop 2021-11-20T06:05:49.917 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.931+0000 7efc4fc0c700 1 -- 172.21.15.139:0/2694673531 shutdown_connections 2021-11-20T06:05:49.917 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.931+0000 7efc4fc0c700 1 --2- 172.21.15.139:0/2694673531 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7efc3405c2a0 0x7efc3405e760 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:49.917 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.931+0000 7efc4fc0c700 1 --2- 172.21.15.139:0/2694673531 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc480f4960 0x7efc48106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:49.918 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.931+0000 7efc4fc0c700 1 --2- 172.21.15.139:0/2694673531 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc480f5300 0x7efc481073e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:49.918 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.931+0000 7efc4fc0c700 1 -- 172.21.15.139:0/2694673531 >> 172.21.15.139:0/2694673531 conn(0x7efc480f0440 msgr2=0x7efc480fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:49.918 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.931+0000 7efc4fc0c700 1 -- 172.21.15.139:0/2694673531 shutdown_connections 2021-11-20T06:05:49.918 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:49.931+0000 7efc4fc0c700 1 -- 172.21.15.139:0/2694673531 wait complete. 2021-11-20T06:05:49.919 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 50 2021-11-20T06:05:49.927 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:05:49.314169+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14560': {'gid': 14560, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 49, 'state': 'up:reconnect', 'state_seq': 5, 'addr': '172.21.15.139:6829/1072553578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 1072553578}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 1072553578}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:05:49.928 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14560:up:reconnect 2021-11-20T06:05:50.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:50 smithi139 conmon[33501]: cluster 2021-11-20T06:05:49.317382+0000 mon.smithi139 2021-11-20T06:05:50.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:50 smithi139 conmon[33501]: (mon.0) 660 : cluster [DBG] mds.? [v2:172.21.15.139:6828/1072553578,v1:172.21.15.139:6829/1072553578] up:reconnect 2021-11-20T06:05:50.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:50 smithi139 conmon[33501]: cluster 2021-11-20T06:05:49.317451+0000 mon.smithi139 (mon.0) 661 2021-11-20T06:05:50.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:50 smithi139 conmon[33501]: : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:reconnect} 3 up:standby 2021-11-20T06:05:50.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:50 smithi139 conmon[33501]: audit 2021-11 2021-11-20T06:05:50.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:50 smithi139 conmon[33501]: -20T06:05:49.929767+0000 mon.smithi139 2021-11-20T06:05:50.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:50 smithi139 conmon[33501]: (mon.0) 662 : audit [DBG] from='client.? 172.21.15.139:0/2694673531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:05:50.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:50 smithi178 conmon[36600]: cluster 2021-11-20T06:05:49.317382+0000 mon.smithi139 (mon.0) 660 : cluster [DBG] mds.? [v2:172.21.15.139:6828/1072553578,v1:172.21.15.139:6829/1072553578] up:reconnect 2021-11-20T06:05:50.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:50 smithi178 conmon[36600]: cluster 2021-11-20T06:05:49.317451+0000 mon.smithi139 (mon.0) 661 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:reconnect} 3 up:standby 2021-11-20T06:05:50.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:50 smithi178 conmon[36600]: audit 2021-11-20T06:05:49.929767+0000 mon.smithi139 (mon.0) 662 : audit [DBG] from='client.? 172.21.15.139:0/2694673531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:05:50.933 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:05:51.056 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.068+0000 7f98ec64b700 1 -- 172.21.15.139:0/3871064895 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98e40f7920 msgr2=0x7f98e40f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:51.057 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.068+0000 7f98ec64b700 1 --2- 172.21.15.139:0/3871064895 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98e40f7920 0x7f98e40f7d80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f98d8004660 tx=0x7f98d8009b30).stop 2021-11-20T06:05:51.057 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.068+0000 7f98ec64b700 1 -- 172.21.15.139:0/3871064895 shutdown_connections 2021-11-20T06:05:51.057 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.068+0000 7f98ec64b700 1 --2- 172.21.15.139:0/3871064895 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98e40f7920 0x7f98e40f7d80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:51.057 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.068+0000 7f98ec64b700 1 --2- 172.21.15.139:0/3871064895 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98e40f6680 0x7f98e40f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:51.057 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.068+0000 7f98ec64b700 1 -- 172.21.15.139:0/3871064895 >> 172.21.15.139:0/3871064895 conn(0x7f98e40f1de0 msgr2=0x7f98e40f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:51.058 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.068+0000 7f98ec64b700 1 -- 172.21.15.139:0/3871064895 shutdown_connections 2021-11-20T06:05:51.058 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.068+0000 7f98ec64b700 1 -- 172.21.15.139:0/3871064895 wait complete. 2021-11-20T06:05:51.058 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.069+0000 7f98ec64b700 1 Processor -- start 2021-11-20T06:05:51.058 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.069+0000 7f98ec64b700 1 -- start start 2021-11-20T06:05:51.059 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.069+0000 7f98ec64b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98e40f6680 0x7f98e4106e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:51.059 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.069+0000 7f98ec64b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98e40f7920 0x7f98e41072f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:51.059 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.069+0000 7f98ec64b700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e41078a0 con 0x7f98e40f6680 2021-11-20T06:05:51.059 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.069+0000 7f98ec64b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e41079e0 con 0x7f98e40f7920 2021-11-20T06:05:51.060 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.069+0000 7f98ea3e7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98e40f6680 0x7f98e4106e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:51.060 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.069+0000 7f98ea3e7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98e40f6680 0x7f98e4106e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57942/0 (socket says 172.21.15.139:57942) 2021-11-20T06:05:51.060 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.069+0000 7f98ea3e7700 1 -- 172.21.15.139:0/385505307 learned_addr learned my addr 172.21.15.139:0/385505307 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:05:51.060 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.069+0000 7f98e9be6700 1 --2- 172.21.15.139:0/385505307 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98e40f7920 0x7f98e41072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:51.060 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.069+0000 7f98ea3e7700 1 -- 172.21.15.139:0/385505307 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98e40f7920 msgr2=0x7f98e41072f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:51.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.069+0000 7f98ea3e7700 1 --2- 172.21.15.139:0/385505307 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98e40f7920 0x7f98e41072f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:51.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.069+0000 7f98ea3e7700 1 -- 172.21.15.139:0/385505307 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d8005040 con 0x7f98e40f6680 2021-11-20T06:05:51.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.069+0000 7f98e9be6700 1 --2- 172.21.15.139:0/385505307 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98e40f7920 0x7f98e41072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:05:51.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.069+0000 7f98ea3e7700 1 --2- 172.21.15.139:0/385505307 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98e40f6680 0x7f98e4106e00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f98d400eb70 tx=0x7f98d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:51.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.070+0000 7f98e2ffd700 1 -- 172.21.15.139:0/385505307 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98d40092e0 con 0x7f98e40f6680 2021-11-20T06:05:51.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.070+0000 7f98e2ffd700 1 -- 172.21.15.139:0/385505307 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98d400f040 con 0x7f98e40f6680 2021-11-20T06:05:51.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.070+0000 7f98ec64b700 1 -- 172.21.15.139:0/385505307 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e410c480 con 0x7f98e40f6680 2021-11-20T06:05:51.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.070+0000 7f98e2ffd700 1 -- 172.21.15.139:0/385505307 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98d400c9a0 con 0x7f98e40f6680 2021-11-20T06:05:51.063 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.070+0000 7f98ec64b700 1 -- 172.21.15.139:0/385505307 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e410ca40 con 0x7f98e40f6680 2021-11-20T06:05:51.063 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.071+0000 7f98ec64b700 1 -- 172.21.15.139:0/385505307 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e4027a00 con 0x7f98e40f6680 2021-11-20T06:05:51.063 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.071+0000 7f98e2ffd700 1 -- 172.21.15.139:0/385505307 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f98d401b030 con 0x7f98e40f6680 2021-11-20T06:05:51.063 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.071+0000 7f98e2ffd700 1 --2- 172.21.15.139:0/385505307 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f98cc05c2f0 0x7f98cc05e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:51.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.072+0000 7f98e2ffd700 1 -- 172.21.15.139:0/385505307 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f98d40763f0 con 0x7f98e40f6680 2021-11-20T06:05:51.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.072+0000 7f98e9be6700 1 --2- 172.21.15.139:0/385505307 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f98cc05c2f0 0x7f98cc05e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:51.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.072+0000 7f98e9be6700 1 --2- 172.21.15.139:0/385505307 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f98cc05c2f0 0x7f98cc05e7b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f98d8005010 tx=0x7f98d800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:51.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.075+0000 7f98e2ffd700 1 -- 172.21.15.139:0/385505307 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98d4048d40 con 0x7f98e40f6680 2021-11-20T06:05:51.266 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.281+0000 7f98ec64b700 1 -- 172.21.15.139:0/385505307 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98e40fee10 con 0x7f98e40f6680 2021-11-20T06:05:51.267 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.282+0000 7f98e2ffd700 1 -- 172.21.15.139:0/385505307 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f98d4019b90 con 0x7f98e40f6680 2021-11-20T06:05:51.267 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:05:51.267 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":48},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":51,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:05:50.315512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14560":{"gid":14560,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.139:6829/1072553578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1072553578},{"type":"v1","addr":"172.21.15.139:6829","nonce":1072553578}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:05:51.268 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.283+0000 7f98ec64b700 1 -- 172.21.15.139:0/385505307 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f98cc05c2f0 msgr2=0x7f98cc05e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:51.269 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.283+0000 7f98ec64b700 1 --2- 172.21.15.139:0/385505307 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f98cc05c2f0 0x7f98cc05e7b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f98d8005010 tx=0x7f98d800b040).stop 2021-11-20T06:05:51.269 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.283+0000 7f98ec64b700 1 -- 172.21.15.139:0/385505307 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98e40f6680 msgr2=0x7f98e4106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:51.269 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.283+0000 7f98ec64b700 1 --2- 172.21.15.139:0/385505307 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98e40f6680 0x7f98e4106e00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f98d400eb70 tx=0x7f98d400b040).stop 2021-11-20T06:05:51.269 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.283+0000 7f98ec64b700 1 -- 172.21.15.139:0/385505307 shutdown_connections 2021-11-20T06:05:51.269 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.283+0000 7f98ec64b700 1 --2- 172.21.15.139:0/385505307 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f98cc05c2f0 0x7f98cc05e7b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:51.270 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.283+0000 7f98ec64b700 1 --2- 172.21.15.139:0/385505307 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f98e40f6680 0x7f98e4106e00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:51.270 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.283+0000 7f98ec64b700 1 --2- 172.21.15.139:0/385505307 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98e40f7920 0x7f98e41072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:51.270 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.283+0000 7f98ec64b700 1 -- 172.21.15.139:0/385505307 >> 172.21.15.139:0/385505307 conn(0x7f98e40f1de0 msgr2=0x7f98e40fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:51.270 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.283+0000 7f98ec64b700 1 -- 172.21.15.139:0/385505307 shutdown_connections 2021-11-20T06:05:51.271 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:51.283+0000 7f98ec64b700 1 -- 172.21.15.139:0/385505307 wait complete. 2021-11-20T06:05:51.271 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 51 2021-11-20T06:05:51.280 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:05:50.315512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14560': {'gid': 14560, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 49, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.139:6829/1072553578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 1072553578}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 1072553578}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:05:51.280 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14560:up:rejoin 2021-11-20T06:05:51.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:51 smithi178 conmon[36600]: cluster 2021-11-20T06:05:50.248388+0000 mgr.smithi139.dplyxn (mgr.14162) 261 : cluster [DBG] pgmap v230: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T06:05:51.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:51 smithi178 conmon[36600]: cluster 2021-11-20T06:05:50 2021-11-20T06:05:51.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:51 smithi178 conmon[36600]: .318386+0000 mon.smithi139 (mon.0) 663 : cluster [DBG] mds.? [v2:172.21.15.139:6828/1072553578,v1:172.21.15.139:6829/1072553578] up:rejoin 2021-11-20T06:05:51.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:51 smithi178 conmon[36600]: cluster 2021-11-20T06:05:50.318446+0000 mon.smithi139 (mon.0) 664 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:05:51.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:51 smithi178 conmon[36600]: audit 2021 2021-11-20T06:05:51.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:51 smithi178 conmon[36600]: -11-20T06:05:51.282440+0000 mon.smithi139 (mon.0) 665 : audit [DBG] from='client.? 172.21.15.139:0/385505307' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:05:51.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:51 smithi178 conmon[36600]: 2021-11-20T06:05:51.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:51 smithi139 conmon[33501]: cluster 2021-11-20T06:05:50.248388+0000 mgr.smithi139.dplyxn (mgr.14162) 261 : cluster [DBG] pgmap v230: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-20T06:05:51.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:51 smithi139 conmon[33501]: cluster 2021-11-20T06: 2021-11-20T06:05:51.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:51 smithi139 conmon[33501]: 05:50.318386+0000 mon.smithi139 (mon.0) 663 : cluster [DBG] mds.? [v2:172.21.15.139:6828/1072553578,v1:172.21.15.139:6829/1072553578] up:rejoin 2021-11-20T06:05:51.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:51 smithi139 conmon[33501]: cluster 2021-11-20T06:05:50.318446+0000 mon.smithi139 (mon.0) 664 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:05:51.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:51 smithi139 conmon[33501]: audit 2021-11-20T06:05:51.282440+0000 mon.smithi139 (mon.0) 665 : audit [DBG] from='client.? 172.21.15.139:0/385505307' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:05:52.281 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:05:52.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.420+0000 7fb3ac651700 1 -- 172.21.15.139:0/3282108448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a40f66b0 msgr2=0x7fb3a40f6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:52.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.420+0000 7fb3ac651700 1 --2- 172.21.15.139:0/3282108448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a40f66b0 0x7fb3a40f6ad0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fb3940099e0 tx=0x7fb394004d10).stop 2021-11-20T06:05:52.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.420+0000 7fb3ac651700 1 -- 172.21.15.139:0/3282108448 shutdown_connections 2021-11-20T06:05:52.409 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.420+0000 7fb3ac651700 1 --2- 172.21.15.139:0/3282108448 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3a40f7950 0x7fb3a40f7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:52.409 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.420+0000 7fb3ac651700 1 --2- 172.21.15.139:0/3282108448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a40f66b0 0x7fb3a40f6ad0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:52.409 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.420+0000 7fb3ac651700 1 -- 172.21.15.139:0/3282108448 >> 172.21.15.139:0/3282108448 conn(0x7fb3a40f1dd0 msgr2=0x7fb3a40f4230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:52.409 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.420+0000 7fb3ac651700 1 -- 172.21.15.139:0/3282108448 shutdown_connections 2021-11-20T06:05:52.410 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.420+0000 7fb3ac651700 1 -- 172.21.15.139:0/3282108448 wait complete. 2021-11-20T06:05:52.410 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3ac651700 1 Processor -- start 2021-11-20T06:05:52.410 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3ac651700 1 -- start start 2021-11-20T06:05:52.410 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3ac651700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3a40f66b0 0x7fb3a4106ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:52.411 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3ac651700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a40f7950 0x7fb3a41074e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:52.411 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3ac651700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a4107a90 con 0x7fb3a40f7950 2021-11-20T06:05:52.411 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3ac651700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a4107bd0 con 0x7fb3a40f66b0 2021-11-20T06:05:52.411 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3a9bec700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a40f7950 0x7fb3a41074e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:52.411 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3a9bec700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a40f7950 0x7fb3a41074e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57960/0 (socket says 172.21.15.139:57960) 2021-11-20T06:05:52.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3a9bec700 1 -- 172.21.15.139:0/2187044417 learned_addr learned my addr 172.21.15.139:0/2187044417 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:05:52.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3a9bec700 1 -- 172.21.15.139:0/2187044417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3a40f66b0 msgr2=0x7fb3a4106ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:52.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3a9bec700 1 --2- 172.21.15.139:0/2187044417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3a40f66b0 0x7fb3a4106ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:05:52.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3a9bec700 1 -- 172.21.15.139:0/2187044417 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb394009640 con 0x7fb3a40f7950 2021-11-20T06:05:52.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3a9bec700 1 --2- 172.21.15.139:0/2187044417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a40f7950 0x7fb3a41074e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fb39800e7f0 tx=0x7fb39800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:52.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3a2ffd700 1 -- 172.21.15.139:0/2187044417 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3980092e0 con 0x7fb3a40f7950 2021-11-20T06:05:52.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3a2ffd700 1 -- 172.21.15.139:0/2187044417 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb39800f040 con 0x7fb3a40f7950 2021-11-20T06:05:52.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3a2ffd700 1 -- 172.21.15.139:0/2187044417 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3980146c0 con 0x7fb3a40f7950 2021-11-20T06:05:52.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.421+0000 7fb3ac651700 1 -- 172.21.15.139:0/2187044417 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a410c670 con 0x7fb3a40f7950 2021-11-20T06:05:52.414 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.422+0000 7fb3ac651700 1 -- 172.21.15.139:0/2187044417 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a410cc30 con 0x7fb3a40f7950 2021-11-20T06:05:52.414 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.422+0000 7fb3ac651700 1 -- 172.21.15.139:0/2187044417 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a4027a00 con 0x7fb3a40f7950 2021-11-20T06:05:52.414 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.423+0000 7fb3a2ffd700 1 -- 172.21.15.139:0/2187044417 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb39801b030 con 0x7fb3a40f7950 2021-11-20T06:05:52.414 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.423+0000 7fb3a2ffd700 1 --2- 172.21.15.139:0/2187044417 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb38c05c2a0 0x7fb38c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:52.415 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.423+0000 7fb3a2ffd700 1 -- 172.21.15.139:0/2187044417 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fb398075f20 con 0x7fb3a40f7950 2021-11-20T06:05:52.415 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.423+0000 7fb3aa3ed700 1 --2- 172.21.15.139:0/2187044417 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb38c05c2a0 0x7fb38c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:52.415 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.423+0000 7fb3aa3ed700 1 --2- 172.21.15.139:0/2187044417 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb38c05c2a0 0x7fb38c05e760 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb394000f80 tx=0x7fb394004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:52.415 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.427+0000 7fb3a2ffd700 1 -- 172.21.15.139:0/2187044417 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3980a22b0 con 0x7fb3a40f7950 2021-11-20T06:05:52.594 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.609+0000 7fb3ac651700 1 -- 172.21.15.139:0/2187044417 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3a40fee40 con 0x7fb3a40f7950 2021-11-20T06:05:52.595 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.610+0000 7fb3a2ffd700 1 -- 172.21.15.139:0/2187044417 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2943 (secure 0 0 0) 0x7fb398019020 con 0x7fb3a40f7950 2021-11-20T06:05:52.596 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:05:52.596 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":52},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":51,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:05:50.315512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14560":{"gid":14560,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.139:6829/1072553578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1072553578},{"type":"v1","addr":"172.21.15.139:6829","nonce":1072553578}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:05:52.597 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.611+0000 7fb3ac651700 1 -- 172.21.15.139:0/2187044417 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb38c05c2a0 msgr2=0x7fb38c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:52.597 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.611+0000 7fb3ac651700 1 --2- 172.21.15.139:0/2187044417 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb38c05c2a0 0x7fb38c05e760 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb394000f80 tx=0x7fb394004d10).stop 2021-11-20T06:05:52.598 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.611+0000 7fb3ac651700 1 -- 172.21.15.139:0/2187044417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a40f7950 msgr2=0x7fb3a41074e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:52.598 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.611+0000 7fb3ac651700 1 --2- 172.21.15.139:0/2187044417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a40f7950 0x7fb3a41074e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fb39800e7f0 tx=0x7fb39800b040).stop 2021-11-20T06:05:52.598 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.611+0000 7fb3ac651700 1 -- 172.21.15.139:0/2187044417 shutdown_connections 2021-11-20T06:05:52.598 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.611+0000 7fb3ac651700 1 --2- 172.21.15.139:0/2187044417 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb38c05c2a0 0x7fb38c05e760 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:52.599 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.611+0000 7fb3ac651700 1 --2- 172.21.15.139:0/2187044417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3a40f66b0 0x7fb3a4106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:05:52.599 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.611+0000 7fb3ac651700 1 --2- 172.21.15.139:0/2187044417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb3a40f7950 0x7fb3a41074e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:52.599 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.611+0000 7fb3ac651700 1 -- 172.21.15.139:0/2187044417 >> 172.21.15.139:0/2187044417 conn(0x7fb3a40f1dd0 msgr2=0x7fb3a40fb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:52.599 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.612+0000 7fb3ac651700 1 -- 172.21.15.139:0/2187044417 shutdown_connections 2021-11-20T06:05:52.600 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:52.612+0000 7fb3ac651700 1 -- 172.21.15.139:0/2187044417 wait complete. 2021-11-20T06:05:52.600 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 52 2021-11-20T06:05:52.608 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:05:50.315512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14560': {'gid': 14560, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 49, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.139:6829/1072553578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 1072553578}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 1072553578}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:05:52.609 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14560:up:rejoin 2021-11-20T06:05:53.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:53 smithi139 conmon[33501]: cluster 2021-11-20T06:05:52.248926+0000 mgr.smithi139.dplyxn (mgr.14162) 262 : cluster 2021-11-20T06:05:53.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:53 smithi139 conmon[33501]: [DBG] pgmap v231: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:53.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:53 smithi139 conmon[33501]: cluster 2021-11-20T06:05:52.325241+0000 mon.smithi139 (mon.0) 666 : cluster [DBG] mds.? [v2:172.21.15.139:6826/3938771458,v1:172.21.15.139:6827/3938771458] up:standby 2021-11-20T06:05:53.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:53 smithi139 conmon[33501]: cluster 2021-11-20T06:05:52.325315 2021-11-20T06:05:53.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:53 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 667 : cluster [DBG] 2021-11-20T06:05:53.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:53 smithi139 conmon[33501]: fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:05:53.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:53 smithi139 conmon[33501]: audit 2021-11-20T 2021-11-20T06:05:53.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:53 smithi139 conmon[33501]: 06:05:52.610787+0000 mon.smithi139 (mon. 2021-11-20T06:05:53.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:53 smithi139 conmon[33501]: 0) 668 : audit [DBG] from='client.? 172.21.15.139:0/2187044417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:05:53.610 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:05:53.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:53 smithi178 conmon[36600]: cluster 2021-11-20T06:05:52.248926+0000 mgr.smithi139.dplyxn (mgr.14162) 262 : cluster [DBG] pgmap v231: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:53.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:53 smithi178 conmon[36600]: cluster 2021-11-20T06:05:52.325241+0000 mon.smithi139 2021-11-20T06:05:53.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:53 smithi178 conmon[36600]: (mon.0) 666 : cluster [DBG] mds.? [v2:172.21.15.139:6826/3938771458,v1:172.21.15.139:6827/3938771458] up:standby 2021-11-20T06:05:53.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:53 smithi178 conmon[36600]: cluster 2021-11-20T06:05:52.325315+0000 mon.smithi139 (mon.0) 667 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:05:53.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:53 smithi178 conmon[36600]: audit 2021-11-20T06:05:52.610787+0000 2021-11-20T06:05:53.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:53 smithi178 conmon[36600]: mon.smithi139 (mon.0) 668 : audit [DBG] from='client.? 172.21.15.139:0/2187044417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:05:53.738 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.751+0000 7f7a64fc6700 1 -- 172.21.15.139:0/2996984742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a600f5580 msgr2=0x7f7a600f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:53.738 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.751+0000 7f7a64fc6700 1 --2- 172.21.15.139:0/2996984742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a600f5580 0x7f7a600f59a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f7a50004660 tx=0x7f7a50009b30).stop 2021-11-20T06:05:53.739 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.751+0000 7f7a64fc6700 1 -- 172.21.15.139:0/2996984742 shutdown_connections 2021-11-20T06:05:53.739 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.751+0000 7f7a64fc6700 1 --2- 172.21.15.139:0/2996984742 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a600f6820 0x7f7a600f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:53.739 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.751+0000 7f7a64fc6700 1 --2- 172.21.15.139:0/2996984742 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a600f5580 0x7f7a600f59a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:53.739 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.751+0000 7f7a64fc6700 1 -- 172.21.15.139:0/2996984742 >> 172.21.15.139:0/2996984742 conn(0x7f7a600f0ce0 msgr2=0x7f7a600f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:53.739 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.751+0000 7f7a64fc6700 1 -- 172.21.15.139:0/2996984742 shutdown_connections 2021-11-20T06:05:53.740 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.751+0000 7f7a64fc6700 1 -- 172.21.15.139:0/2996984742 wait complete. 2021-11-20T06:05:53.740 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.752+0000 7f7a64fc6700 1 Processor -- start 2021-11-20T06:05:53.740 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.752+0000 7f7a64fc6700 1 -- start start 2021-11-20T06:05:53.740 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.752+0000 7f7a64fc6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a600f5580 0x7f7a60105db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:53.741 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.752+0000 7f7a64fc6700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a600f6820 0x7f7a601062a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:53.741 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.752+0000 7f7a64fc6700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a60106850 con 0x7f7a600f5580 2021-11-20T06:05:53.741 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.752+0000 7f7a64fc6700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a60106990 con 0x7f7a600f6820 2021-11-20T06:05:53.741 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.752+0000 7f7a5e59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a600f5580 0x7f7a60105db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:53.741 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.752+0000 7f7a5e59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a600f5580 0x7f7a60105db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57970/0 (socket says 172.21.15.139:57970) 2021-11-20T06:05:53.742 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.752+0000 7f7a5e59c700 1 -- 172.21.15.139:0/174585893 learned_addr learned my addr 172.21.15.139:0/174585893 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:05:53.742 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.752+0000 7f7a5dd9b700 1 --2- 172.21.15.139:0/174585893 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a600f6820 0x7f7a601062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:53.742 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.752+0000 7f7a5e59c700 1 -- 172.21.15.139:0/174585893 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a600f6820 msgr2=0x7f7a601062a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:53.742 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.752+0000 7f7a5e59c700 1 --2- 172.21.15.139:0/174585893 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a600f6820 0x7f7a601062a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:53.743 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.752+0000 7f7a5e59c700 1 -- 172.21.15.139:0/174585893 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a50005040 con 0x7f7a600f5580 2021-11-20T06:05:53.743 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.752+0000 7f7a5e59c700 1 --2- 172.21.15.139:0/174585893 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a600f5580 0x7f7a60105db0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f7a50004e30 tx=0x7f7a50009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:53.743 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.752+0000 7f7a4effd700 1 -- 172.21.15.139:0/174585893 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a50018070 con 0x7f7a600f5580 2021-11-20T06:05:53.743 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.753+0000 7f7a4effd700 1 -- 172.21.15.139:0/174585893 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a50013410 con 0x7f7a600f5580 2021-11-20T06:05:53.743 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.753+0000 7f7a64fc6700 1 -- 172.21.15.139:0/174585893 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6010b3d0 con 0x7f7a600f5580 2021-11-20T06:05:53.744 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.753+0000 7f7a4effd700 1 -- 172.21.15.139:0/174585893 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a5001de20 con 0x7f7a600f5580 2021-11-20T06:05:53.744 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.753+0000 7f7a64fc6700 1 -- 172.21.15.139:0/174585893 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6010b990 con 0x7f7a600f5580 2021-11-20T06:05:53.745 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.754+0000 7f7a4effd700 1 -- 172.21.15.139:0/174585893 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7a5001d420 con 0x7f7a600f5580 2021-11-20T06:05:53.745 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.754+0000 7f7a64fc6700 1 -- 172.21.15.139:0/174585893 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a60027a00 con 0x7f7a600f5580 2021-11-20T06:05:53.745 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.754+0000 7f7a4effd700 1 --2- 172.21.15.139:0/174585893 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7a4805c2a0 0x7f7a4805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:53.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.754+0000 7f7a4effd700 1 -- 172.21.15.139:0/174585893 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f7a50077cd0 con 0x7f7a600f5580 2021-11-20T06:05:53.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.754+0000 7f7a5dd9b700 1 --2- 172.21.15.139:0/174585893 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7a4805c2a0 0x7f7a4805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:53.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.754+0000 7f7a5dd9b700 1 --2- 172.21.15.139:0/174585893 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7a4805c2a0 0x7f7a4805e760 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f7a54004c40 tx=0x7f7a5400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:53.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.758+0000 7f7a4effd700 1 -- 172.21.15.139:0/174585893 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a5004a760 con 0x7f7a600f5580 2021-11-20T06:05:53.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.941+0000 7f7a64fc6700 1 -- 172.21.15.139:0/174585893 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a600fdd10 con 0x7f7a600f5580 2021-11-20T06:05:53.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.941+0000 7f7a4effd700 1 -- 172.21.15.139:0/174585893 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2943 (secure 0 0 0) 0x7f7a5004dd80 con 0x7f7a600f5580 2021-11-20T06:05:53.927 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:05:53.927 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":52},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":51,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:05:50.315512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14560":{"gid":14560,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.139:6829/1072553578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1072553578},{"type":"v1","addr":"172.21.15.139:6829","nonce":1072553578}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:05:53.929 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.943+0000 7f7a64fc6700 1 -- 172.21.15.139:0/174585893 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7a4805c2a0 msgr2=0x7f7a4805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:53.929 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.943+0000 7f7a64fc6700 1 --2- 172.21.15.139:0/174585893 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7a4805c2a0 0x7f7a4805e760 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f7a54004c40 tx=0x7f7a5400b040).stop 2021-11-20T06:05:53.929 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.943+0000 7f7a64fc6700 1 -- 172.21.15.139:0/174585893 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a600f5580 msgr2=0x7f7a60105db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:53.930 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.943+0000 7f7a64fc6700 1 --2- 172.21.15.139:0/174585893 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a600f5580 0x7f7a60105db0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f7a50004e30 tx=0x7f7a50009b30).stop 2021-11-20T06:05:53.930 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.943+0000 7f7a64fc6700 1 -- 172.21.15.139:0/174585893 shutdown_connections 2021-11-20T06:05:53.930 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.943+0000 7f7a64fc6700 1 --2- 172.21.15.139:0/174585893 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7a4805c2a0 0x7f7a4805e760 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:53.931 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.943+0000 7f7a64fc6700 1 --2- 172.21.15.139:0/174585893 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a600f5580 0x7f7a60105db0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:53.931 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.943+0000 7f7a64fc6700 1 --2- 172.21.15.139:0/174585893 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a600f6820 0x7f7a601062a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:53.931 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.943+0000 7f7a64fc6700 1 -- 172.21.15.139:0/174585893 >> 172.21.15.139:0/174585893 conn(0x7f7a600f0ce0 msgr2=0x7f7a600fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:53.931 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.943+0000 7f7a64fc6700 1 -- 172.21.15.139:0/174585893 shutdown_connections 2021-11-20T06:05:53.931 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:53.943+0000 7f7a64fc6700 1 -- 172.21.15.139:0/174585893 wait complete. 2021-11-20T06:05:53.932 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 52 2021-11-20T06:05:53.939 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:05:50.315512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14560': {'gid': 14560, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 49, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.139:6829/1072553578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 1072553578}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 1072553578}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:05:53.940 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14560:up:rejoin 2021-11-20T06:05:54.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:54 smithi178 conmon[36600]: audit 2021-11-20T06:05:53.942087+0000 mon.smithi139 (mon.0) 669 : audit [DBG] from='client.? 172.21.15.139:0/174585893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:05:54.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:54 smithi139 conmon[33501]: audit 2021-11-20T06:05:53.942087+0000 mon.smithi139 (mon.0) 669 : audit 2021-11-20T06:05:54.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:54 smithi139 conmon[33501]: [DBG] from='client.? 172.21.15.139:0/174585893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:05:54.942 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:05:55.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.080+0000 7f38f7185700 1 -- 172.21.15.139:0/1214746735 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38f00f6820 msgr2=0x7f38f00f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:55.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.080+0000 7f38f7185700 1 --2- 172.21.15.139:0/1214746735 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38f00f6820 0x7f38f00f6c80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f38dc004660 tx=0x7f38dc009b30).stop 2021-11-20T06:05:55.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.081+0000 7f38f7185700 1 -- 172.21.15.139:0/1214746735 shutdown_connections 2021-11-20T06:05:55.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.081+0000 7f38f7185700 1 --2- 172.21.15.139:0/1214746735 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38f00f6820 0x7f38f00f6c80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:55.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.081+0000 7f38f7185700 1 --2- 172.21.15.139:0/1214746735 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38f00f5580 0x7f38f00f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:55.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.081+0000 7f38f7185700 1 -- 172.21.15.139:0/1214746735 >> 172.21.15.139:0/1214746735 conn(0x7f38f00f0ce0 msgr2=0x7f38f00f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:55.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.081+0000 7f38f7185700 1 -- 172.21.15.139:0/1214746735 shutdown_connections 2021-11-20T06:05:55.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.081+0000 7f38f7185700 1 -- 172.21.15.139:0/1214746735 wait complete. 2021-11-20T06:05:55.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.081+0000 7f38f7185700 1 Processor -- start 2021-11-20T06:05:55.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.081+0000 7f38f7185700 1 -- start start 2021-11-20T06:05:55.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.081+0000 7f38f7185700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38f00f5580 0x7f38f0105dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:55.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.081+0000 7f38f7185700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38f00f6820 0x7f38f01062b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:55.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.081+0000 7f38f7185700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f0106860 con 0x7f38f00f6820 2021-11-20T06:05:55.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.081+0000 7f38f7185700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f01069a0 con 0x7f38f00f5580 2021-11-20T06:05:55.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.082+0000 7f38effff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38f00f6820 0x7f38f01062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:55.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.082+0000 7f38effff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38f00f6820 0x7f38f01062b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57982/0 (socket says 172.21.15.139:57982) 2021-11-20T06:05:55.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.082+0000 7f38effff700 1 -- 172.21.15.139:0/1660823279 learned_addr learned my addr 172.21.15.139:0/1660823279 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:05:55.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.082+0000 7f38f4f21700 1 --2- 172.21.15.139:0/1660823279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38f00f5580 0x7f38f0105dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:55.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.082+0000 7f38effff700 1 -- 172.21.15.139:0/1660823279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38f00f5580 msgr2=0x7f38f0105dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:55.072 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.082+0000 7f38effff700 1 --2- 172.21.15.139:0/1660823279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38f00f5580 0x7f38f0105dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:55.072 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.082+0000 7f38effff700 1 -- 172.21.15.139:0/1660823279 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38dc005040 con 0x7f38f00f6820 2021-11-20T06:05:55.072 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.082+0000 7f38effff700 1 --2- 172.21.15.139:0/1660823279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38f00f6820 0x7f38f01062b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f38dc009830 tx=0x7f38dc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:55.072 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.082+0000 7f38ed7fa700 1 -- 172.21.15.139:0/1660823279 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38dc018070 con 0x7f38f00f6820 2021-11-20T06:05:55.073 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.082+0000 7f38f7185700 1 -- 172.21.15.139:0/1660823279 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f010b3e0 con 0x7f38f00f6820 2021-11-20T06:05:55.073 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.082+0000 7f38ed7fa700 1 -- 172.21.15.139:0/1660823279 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38dc013410 con 0x7f38f00f6820 2021-11-20T06:05:55.073 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.082+0000 7f38ed7fa700 1 -- 172.21.15.139:0/1660823279 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38dc01c620 con 0x7f38f00f6820 2021-11-20T06:05:55.074 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.082+0000 7f38f7185700 1 -- 172.21.15.139:0/1660823279 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f010b9a0 con 0x7f38f00f6820 2021-11-20T06:05:55.075 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.083+0000 7f38f7185700 1 -- 172.21.15.139:0/1660823279 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f0027a00 con 0x7f38f00f6820 2021-11-20T06:05:55.075 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.084+0000 7f38ed7fa700 1 -- 172.21.15.139:0/1660823279 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f38dc01c8e0 con 0x7f38f00f6820 2021-11-20T06:05:55.075 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.084+0000 7f38ed7fa700 1 --2- 172.21.15.139:0/1660823279 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f38e005c2a0 0x7f38e005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:55.076 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.084+0000 7f38ed7fa700 1 -- 172.21.15.139:0/1660823279 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f38dc078070 con 0x7f38f00f6820 2021-11-20T06:05:55.076 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.084+0000 7f38f4f21700 1 --2- 172.21.15.139:0/1660823279 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f38e005c2a0 0x7f38e005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:55.076 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.084+0000 7f38f4f21700 1 --2- 172.21.15.139:0/1660823279 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f38e005c2a0 0x7f38e005e760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f38f00f6680 tx=0x7f38e4009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:55.076 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.088+0000 7f38ed7fa700 1 -- 172.21.15.139:0/1660823279 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38dc04aa20 con 0x7f38f00f6820 2021-11-20T06:05:55.278 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.294+0000 7f38f7185700 1 -- 172.21.15.139:0/1660823279 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38f00fdd10 con 0x7f38f00f6820 2021-11-20T06:05:55.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.294+0000 7f38ed7fa700 1 -- 172.21.15.139:0/1660823279 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2943 (secure 0 0 0) 0x7f38dc04e040 con 0x7f38f00f6820 2021-11-20T06:05:55.280 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:05:55.280 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":52},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":51,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:05:50.315512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14560":{"gid":14560,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.139:6829/1072553578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1072553578},{"type":"v1","addr":"172.21.15.139:6829","nonce":1072553578}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:05:55.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.296+0000 7f38f7185700 1 -- 172.21.15.139:0/1660823279 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f38e005c2a0 msgr2=0x7f38e005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:55.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.296+0000 7f38f7185700 1 --2- 172.21.15.139:0/1660823279 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f38e005c2a0 0x7f38e005e760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f38f00f6680 tx=0x7f38e4009040).stop 2021-11-20T06:05:55.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.296+0000 7f38f7185700 1 -- 172.21.15.139:0/1660823279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38f00f6820 msgr2=0x7f38f01062b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:55.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.296+0000 7f38f7185700 1 --2- 172.21.15.139:0/1660823279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38f00f6820 0x7f38f01062b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f38dc009830 tx=0x7f38dc009b30).stop 2021-11-20T06:05:55.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.296+0000 7f38f7185700 1 -- 172.21.15.139:0/1660823279 shutdown_connections 2021-11-20T06:05:55.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.296+0000 7f38f7185700 1 --2- 172.21.15.139:0/1660823279 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f38e005c2a0 0x7f38e005e760 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:55.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.296+0000 7f38f7185700 1 --2- 172.21.15.139:0/1660823279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38f00f5580 0x7f38f0105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:55.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.296+0000 7f38f7185700 1 --2- 172.21.15.139:0/1660823279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38f00f6820 0x7f38f01062b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:55.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.296+0000 7f38f7185700 1 -- 172.21.15.139:0/1660823279 >> 172.21.15.139:0/1660823279 conn(0x7f38f00f0ce0 msgr2=0x7f38f00fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:55.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.296+0000 7f38f7185700 1 -- 172.21.15.139:0/1660823279 shutdown_connections 2021-11-20T06:05:55.284 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:55.296+0000 7f38f7185700 1 -- 172.21.15.139:0/1660823279 wait complete. 2021-11-20T06:05:55.284 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 52 2021-11-20T06:05:55.293 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:05:50.315512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14560': {'gid': 14560, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 49, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.139:6829/1072553578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 1072553578}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 1072553578}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:05:55.293 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14560:up:rejoin 2021-11-20T06:05:55.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:55 smithi178 conmon[36600]: cluster 2021-11-20T06:05:54.249293+0000 mgr.smithi139.dplyxn (mgr 2021-11-20T06:05:55.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:55 smithi178 conmon[36600]: .14162) 263 : cluster [DBG] pgmap v232: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:55.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:55 smithi178 conmon[36600]: audit 2021-11-20T06:05:55.294950+0000 mon.smithi139 (mon.0) 670 : audit [DBG] from='client.? 172.21.15.139:0/1660823279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:05:55.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:55 smithi139 conmon[33501]: cluster 2021-11-20T06:05:54.249293+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:05:55.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:55 smithi139 conmon[33501]: 263 : cluster [DBG] pgmap v232: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:55.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:55 smithi139 conmon[33501]: audit 2021-11-20T06:05:55.294950+0000 mon.smithi139 (mon.0) 670 : 2021-11-20T06:05:55.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:55 smithi139 conmon[33501]: audit [DBG] from='client.? 172.21.15.139:0/1660823279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:05:56.294 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:05:56.420 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.434+0000 7f32413e7700 1 -- 172.21.15.139:0/3380324187 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f323c0f54d0 msgr2=0x7f323c0f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:56.421 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.434+0000 7f32413e7700 1 --2- 172.21.15.139:0/3380324187 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f323c0f54d0 0x7f323c0f58f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f322c004660 tx=0x7f322c009b30).stop 2021-11-20T06:05:56.421 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.434+0000 7f32413e7700 1 -- 172.21.15.139:0/3380324187 shutdown_connections 2021-11-20T06:05:56.421 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.434+0000 7f32413e7700 1 --2- 172.21.15.139:0/3380324187 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f323c0f6770 0x7f323c0f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:56.421 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.434+0000 7f32413e7700 1 --2- 172.21.15.139:0/3380324187 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f323c0f54d0 0x7f323c0f58f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:56.422 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.434+0000 7f32413e7700 1 -- 172.21.15.139:0/3380324187 >> 172.21.15.139:0/3380324187 conn(0x7f323c0f0c30 msgr2=0x7f323c0f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:56.422 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.434+0000 7f32413e7700 1 -- 172.21.15.139:0/3380324187 shutdown_connections 2021-11-20T06:05:56.422 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.434+0000 7f32413e7700 1 -- 172.21.15.139:0/3380324187 wait complete. 2021-11-20T06:05:56.422 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.434+0000 7f32413e7700 1 Processor -- start 2021-11-20T06:05:56.423 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.434+0000 7f32413e7700 1 -- start start 2021-11-20T06:05:56.423 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f32413e7700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f323c0f54d0 0x7f323c106dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:56.423 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f32413e7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f323c0f6770 0x7f323c1072c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:56.423 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f32413e7700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c1078c0 con 0x7f323c0f6770 2021-11-20T06:05:56.424 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f32413e7700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c108100 con 0x7f323c0f54d0 2021-11-20T06:05:56.424 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f323b7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f323c0f6770 0x7f323c1072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:56.424 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f323b7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f323c0f6770 0x7f323c1072c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:57994/0 (socket says 172.21.15.139:57994) 2021-11-20T06:05:56.424 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f323b7fe700 1 -- 172.21.15.139:0/900606252 learned_addr learned my addr 172.21.15.139:0/900606252 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:05:56.424 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f323bfff700 1 --2- 172.21.15.139:0/900606252 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f323c0f54d0 0x7f323c106dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:56.425 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f323b7fe700 1 -- 172.21.15.139:0/900606252 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f323c0f54d0 msgr2=0x7f323c106dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:56.425 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f323b7fe700 1 --2- 172.21.15.139:0/900606252 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f323c0f54d0 0x7f323c106dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:56.425 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f323b7fe700 1 -- 172.21.15.139:0/900606252 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c005040 con 0x7f323c0f6770 2021-11-20T06:05:56.425 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f323b7fe700 1 --2- 172.21.15.139:0/900606252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f323c0f6770 0x7f323c1072c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f323000c530 tx=0x7f3230007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:56.426 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f3238ff9700 1 -- 172.21.15.139:0/900606252 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f323000f650 con 0x7f323c0f6770 2021-11-20T06:05:56.426 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f3238ff9700 1 -- 172.21.15.139:0/900606252 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f323000fc60 con 0x7f323c0f6770 2021-11-20T06:05:56.426 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f32413e7700 1 -- 172.21.15.139:0/900606252 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323c1083e0 con 0x7f323c0f6770 2021-11-20T06:05:56.426 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f3238ff9700 1 -- 172.21.15.139:0/900606252 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3230012c80 con 0x7f323c0f6770 2021-11-20T06:05:56.427 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.435+0000 7f32413e7700 1 -- 172.21.15.139:0/900606252 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323c108920 con 0x7f323c0f6770 2021-11-20T06:05:56.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.436+0000 7f32413e7700 1 -- 172.21.15.139:0/900606252 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323c0f48c0 con 0x7f323c0f6770 2021-11-20T06:05:56.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.437+0000 7f3238ff9700 1 -- 172.21.15.139:0/900606252 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3230010070 con 0x7f323c0f6770 2021-11-20T06:05:56.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.437+0000 7f3238ff9700 1 --2- 172.21.15.139:0/900606252 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f322405c300 0x7f322405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:56.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.437+0000 7f3238ff9700 1 -- 172.21.15.139:0/900606252 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f3230075ef0 con 0x7f323c0f6770 2021-11-20T06:05:56.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.437+0000 7f323bfff700 1 --2- 172.21.15.139:0/900606252 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f322405c300 0x7f322405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:56.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.437+0000 7f323bfff700 1 --2- 172.21.15.139:0/900606252 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f322405c300 0x7f322405e7c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f323c107b90 tx=0x7f322c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:56.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.441+0000 7f3238ff9700 1 -- 172.21.15.139:0/900606252 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32300452b0 con 0x7f323c0f6770 2021-11-20T06:05:56.610 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.626+0000 7f32413e7700 1 -- 172.21.15.139:0/900606252 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f323c0fedf0 con 0x7f323c0f6770 2021-11-20T06:05:56.611 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.626+0000 7f3238ff9700 1 -- 172.21.15.139:0/900606252 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2943 (secure 0 0 0) 0x7f323004c330 con 0x7f323c0f6770 2021-11-20T06:05:56.611 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:05:56.612 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":52},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":51,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:05:50.315512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14560":{"gid":14560,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.139:6829/1072553578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1072553578},{"type":"v1","addr":"172.21.15.139:6829","nonce":1072553578}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:05:56.613 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.628+0000 7f32413e7700 1 -- 172.21.15.139:0/900606252 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f322405c300 msgr2=0x7f322405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:56.613 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.628+0000 7f32413e7700 1 --2- 172.21.15.139:0/900606252 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f322405c300 0x7f322405e7c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f323c107b90 tx=0x7f322c009b30).stop 2021-11-20T06:05:56.613 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.628+0000 7f32413e7700 1 -- 172.21.15.139:0/900606252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f323c0f6770 msgr2=0x7f323c1072c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:56.614 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.628+0000 7f32413e7700 1 --2- 172.21.15.139:0/900606252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f323c0f6770 0x7f323c1072c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f323000c530 tx=0x7f3230007e20).stop 2021-11-20T06:05:56.614 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.628+0000 7f32413e7700 1 -- 172.21.15.139:0/900606252 shutdown_connections 2021-11-20T06:05:56.614 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.628+0000 7f32413e7700 1 --2- 172.21.15.139:0/900606252 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f322405c300 0x7f322405e7c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:56.614 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.628+0000 7f32413e7700 1 --2- 172.21.15.139:0/900606252 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f323c0f54d0 0x7f323c106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:56.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.628+0000 7f32413e7700 1 --2- 172.21.15.139:0/900606252 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f323c0f6770 0x7f323c1072c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:56.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.628+0000 7f32413e7700 1 -- 172.21.15.139:0/900606252 >> 172.21.15.139:0/900606252 conn(0x7f323c0f0c30 msgr2=0x7f323c0fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:56.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.628+0000 7f32413e7700 1 -- 172.21.15.139:0/900606252 shutdown_connections 2021-11-20T06:05:56.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:56.628+0000 7f32413e7700 1 -- 172.21.15.139:0/900606252 wait complete. 2021-11-20T06:05:56.617 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 52 2021-11-20T06:05:56.625 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:05:50.315512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14560': {'gid': 14560, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 49, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.139:6829/1072553578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 1072553578}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 1072553578}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:05:56.626 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14560:up:rejoin 2021-11-20T06:05:57.628 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:05:57.655 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:57 smithi139 conmon[33501]: cluster 2021-11-20T06:05:56.249817+0000 mgr.smithi139.dplyxn (mgr.14162) 264 : cluster [DBG] pgmap v233: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:57.655 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:57 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:05:57.656 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:57 smithi139 conmon[33501]: 20T06:05:56.627116+0000 mon.smithi139 (mon.0) 671 : audit [DBG] from='client.? 172.21.15.139:0/900606252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:05:57.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:57 smithi178 conmon[36600]: cluster 2021-11-20T06:05:56.249817+0000 mgr.smithi139.dplyxn (mgr.14162) 264 : cluster [DBG] 2021-11-20T06:05:57.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:57 smithi178 conmon[36600]: pgmap v233: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:05:57.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:57 smithi178 conmon[36600]: audit 2021-11-20T06:05:56.627116+0000 mon.smithi139 (mon.0) 671 : audit [DBG] from='client.? 172.21.15.139:0/900606252' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:05:57.756 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.769+0000 7f14d81b8700 1 -- 172.21.15.139:0/3898129416 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14d00f7600 msgr2=0x7f14d00f9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:57.756 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.769+0000 7f14d81b8700 1 --2- 172.21.15.139:0/3898129416 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14d00f7600 0x7f14d00f9a90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f14c4004660 tx=0x7f14c4009b30).stop 2021-11-20T06:05:57.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.769+0000 7f14d81b8700 1 -- 172.21.15.139:0/3898129416 shutdown_connections 2021-11-20T06:05:57.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.769+0000 7f14d81b8700 1 --2- 172.21.15.139:0/3898129416 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14d00f7600 0x7f14d00f9a90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:57.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.769+0000 7f14d81b8700 1 --2- 172.21.15.139:0/3898129416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14d0060750 0x7f14d00f7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:57.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.769+0000 7f14d81b8700 1 -- 172.21.15.139:0/3898129416 >> 172.21.15.139:0/3898129416 conn(0x7f14d00f0d10 msgr2=0x7f14d00f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:57.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.769+0000 7f14d81b8700 1 -- 172.21.15.139:0/3898129416 shutdown_connections 2021-11-20T06:05:57.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.769+0000 7f14d81b8700 1 -- 172.21.15.139:0/3898129416 wait complete. 2021-11-20T06:05:57.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.770+0000 7f14d81b8700 1 Processor -- start 2021-11-20T06:05:57.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.770+0000 7f14d81b8700 1 -- start start 2021-11-20T06:05:57.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.770+0000 7f14d81b8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14d0060750 0x7f14d010d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:57.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.770+0000 7f14d81b8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14d00f7600 0x7f14d010d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:57.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.770+0000 7f14d81b8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d010dee0 con 0x7f14d0060750 2021-11-20T06:05:57.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.770+0000 7f14d81b8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d010e020 con 0x7f14d00f7600 2021-11-20T06:05:57.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.770+0000 7f14d5f54700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14d0060750 0x7f14d010d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:57.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.770+0000 7f14d5f54700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14d0060750 0x7f14d010d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58002/0 (socket says 172.21.15.139:58002) 2021-11-20T06:05:57.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.770+0000 7f14d5f54700 1 -- 172.21.15.139:0/670434567 learned_addr learned my addr 172.21.15.139:0/670434567 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:05:57.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.770+0000 7f14d5f54700 1 -- 172.21.15.139:0/670434567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14d00f7600 msgr2=0x7f14d010d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:57.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.770+0000 7f14d5753700 1 --2- 172.21.15.139:0/670434567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14d00f7600 0x7f14d010d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.770+0000 7f14d5f54700 1 --2- 172.21.15.139:0/670434567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14d00f7600 0x7f14d010d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.770+0000 7f14d5f54700 1 -- 172.21.15.139:0/670434567 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c4005040 con 0x7f14d0060750 2021-11-20T06:05:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.770+0000 7f14d5f54700 1 --2- 172.21.15.139:0/670434567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14d0060750 0x7f14d010d440 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f14c000c7c0 tx=0x7f14c0004d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.770+0000 7f14ce7fc700 1 -- 172.21.15.139:0/670434567 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14c000a6e0 con 0x7f14d0060750 2021-11-20T06:05:57.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.771+0000 7f14ce7fc700 1 -- 172.21.15.139:0/670434567 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14c000acf0 con 0x7f14d0060750 2021-11-20T06:05:57.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.771+0000 7f14d81b8700 1 -- 172.21.15.139:0/670434567 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d0112ac0 con 0x7f14d0060750 2021-11-20T06:05:57.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.771+0000 7f14ce7fc700 1 -- 172.21.15.139:0/670434567 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14c0012df0 con 0x7f14d0060750 2021-11-20T06:05:57.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.771+0000 7f14d81b8700 1 -- 172.21.15.139:0/670434567 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d0113000 con 0x7f14d0060750 2021-11-20T06:05:57.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.771+0000 7f14d81b8700 1 -- 172.21.15.139:0/670434567 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d0027a00 con 0x7f14d0060750 2021-11-20T06:05:57.763 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.772+0000 7f14ce7fc700 1 -- 172.21.15.139:0/670434567 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f14c0019030 con 0x7f14d0060750 2021-11-20T06:05:57.764 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.772+0000 7f14ce7fc700 1 --2- 172.21.15.139:0/670434567 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f14b805c2a0 0x7f14b805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:57.764 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.772+0000 7f14ce7fc700 1 -- 172.21.15.139:0/670434567 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f14c0076320 con 0x7f14d0060750 2021-11-20T06:05:57.764 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.772+0000 7f14d5753700 1 --2- 172.21.15.139:0/670434567 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f14b805c2a0 0x7f14b805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:57.764 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.773+0000 7f14d5753700 1 --2- 172.21.15.139:0/670434567 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f14b805c2a0 0x7f14b805e760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f14c4009d60 tx=0x7f14c4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:57.765 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.776+0000 7f14ce7fc700 1 -- 172.21.15.139:0/670434567 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14c00462b0 con 0x7f14d0060750 2021-11-20T06:05:57.945 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.960+0000 7f14d81b8700 1 -- 172.21.15.139:0/670434567 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14d0105430 con 0x7f14d0060750 2021-11-20T06:05:57.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.961+0000 7f14ce7fc700 1 -- 172.21.15.139:0/670434567 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2943 (secure 0 0 0) 0x7f14c0017020 con 0x7f14d0060750 2021-11-20T06:05:57.946 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:05:57.949 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":52},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":51,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:05:50.315512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14560":{"gid":14560,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.139:6829/1072553578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1072553578},{"type":"v1","addr":"172.21.15.139:6829","nonce":1072553578}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:05:57.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.962+0000 7f14d81b8700 1 -- 172.21.15.139:0/670434567 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f14b805c2a0 msgr2=0x7f14b805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:57.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.962+0000 7f14d81b8700 1 --2- 172.21.15.139:0/670434567 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f14b805c2a0 0x7f14b805e760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f14c4009d60 tx=0x7f14c4009b30).stop 2021-11-20T06:05:57.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.962+0000 7f14d81b8700 1 -- 172.21.15.139:0/670434567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14d0060750 msgr2=0x7f14d010d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:57.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.962+0000 7f14d81b8700 1 --2- 172.21.15.139:0/670434567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14d0060750 0x7f14d010d440 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f14c000c7c0 tx=0x7f14c0004d70).stop 2021-11-20T06:05:57.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.962+0000 7f14d81b8700 1 -- 172.21.15.139:0/670434567 shutdown_connections 2021-11-20T06:05:57.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.962+0000 7f14d81b8700 1 --2- 172.21.15.139:0/670434567 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f14b805c2a0 0x7f14b805e760 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:57.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.962+0000 7f14d81b8700 1 --2- 172.21.15.139:0/670434567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f14d0060750 0x7f14d010d440 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:57.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.962+0000 7f14d81b8700 1 --2- 172.21.15.139:0/670434567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14d00f7600 0x7f14d010d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:57.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.963+0000 7f14d81b8700 1 -- 172.21.15.139:0/670434567 >> 172.21.15.139:0/670434567 conn(0x7f14d00f0d10 msgr2=0x7f14d00f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:57.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.963+0000 7f14d81b8700 1 -- 172.21.15.139:0/670434567 shutdown_connections 2021-11-20T06:05:57.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:57.963+0000 7f14d81b8700 1 -- 172.21.15.139:0/670434567 wait complete. 2021-11-20T06:05:57.954 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 52 2021-11-20T06:05:57.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:05:50.315512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14560': {'gid': 14560, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 49, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.139:6829/1072553578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 1072553578}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 1072553578}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:05:57.960 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14560:up:rejoin 2021-11-20T06:05:58.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:58 smithi178 conmon[36600]: audit 2021-11-20T06:05:57.961763+0000 mon.smithi139 (mon.0) 672 : audit [DBG] from='client.? 172.21.15.139:0/670434567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:05:58.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:58 smithi178 conmon[36600]: 2021-11-20T06:05:58.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:58 smithi139 conmon[33501]: audit 2021-11-20T06:05:57.961763+0000 mon.smithi139 (mon.0) 672 : audit [DBG] 2021-11-20T06:05:58.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:58 smithi139 conmon[33501]: from='client.? 172.21.15.139:0/670434567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:05:58.961 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:05:59.089 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.102+0000 7f72b900c700 1 -- 172.21.15.139:0/4185254078 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72b40f7920 msgr2=0x7f72b40f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:59.089 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.102+0000 7f72b900c700 1 --2- 172.21.15.139:0/4185254078 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72b40f7920 0x7f72b40f7d80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f72a8004660 tx=0x7f72a8009b30).stop 2021-11-20T06:05:59.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.102+0000 7f72b900c700 1 -- 172.21.15.139:0/4185254078 shutdown_connections 2021-11-20T06:05:59.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.102+0000 7f72b900c700 1 --2- 172.21.15.139:0/4185254078 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72b40f7920 0x7f72b40f7d80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:59.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.102+0000 7f72b900c700 1 --2- 172.21.15.139:0/4185254078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72b40f6680 0x7f72b40f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:59.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.102+0000 7f72b900c700 1 -- 172.21.15.139:0/4185254078 >> 172.21.15.139:0/4185254078 conn(0x7f72b40f1de0 msgr2=0x7f72b40f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:59.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.102+0000 7f72b900c700 1 -- 172.21.15.139:0/4185254078 shutdown_connections 2021-11-20T06:05:59.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.102+0000 7f72b900c700 1 -- 172.21.15.139:0/4185254078 wait complete. 2021-11-20T06:05:59.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.102+0000 7f72b900c700 1 Processor -- start 2021-11-20T06:05:59.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.102+0000 7f72b900c700 1 -- start start 2021-11-20T06:05:59.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.102+0000 7f72b900c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72b40f6680 0x7f72b4106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:59.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.102+0000 7f72b900c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72b40f7920 0x7f72b4107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:59.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.102+0000 7f72b900c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b4107930 con 0x7f72b40f7920 2021-11-20T06:05:59.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.102+0000 7f72b900c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b4107a70 con 0x7f72b40f6680 2021-11-20T06:05:59.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.103+0000 7f72b259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72b40f7920 0x7f72b4107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:59.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.103+0000 7f72b259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72b40f7920 0x7f72b4107380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58012/0 (socket says 172.21.15.139:58012) 2021-11-20T06:05:59.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.103+0000 7f72b259c700 1 -- 172.21.15.139:0/1030361805 learned_addr learned my addr 172.21.15.139:0/1030361805 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:05:59.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.103+0000 7f72b2d9d700 1 --2- 172.21.15.139:0/1030361805 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72b40f6680 0x7f72b4106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:59.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.103+0000 7f72b259c700 1 -- 172.21.15.139:0/1030361805 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72b40f6680 msgr2=0x7f72b4106e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:59.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.103+0000 7f72b259c700 1 --2- 172.21.15.139:0/1030361805 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72b40f6680 0x7f72b4106e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:59.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.103+0000 7f72b259c700 1 -- 172.21.15.139:0/1030361805 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a8005040 con 0x7f72b40f7920 2021-11-20T06:05:59.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.103+0000 7f72b259c700 1 --2- 172.21.15.139:0/1030361805 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72b40f7920 0x7f72b4107380 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f72a8009f50 tx=0x7f72a8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:59.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.103+0000 7f729b7fe700 1 -- 172.21.15.139:0/1030361805 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72a8018070 con 0x7f72b40f7920 2021-11-20T06:05:59.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.103+0000 7f729b7fe700 1 -- 172.21.15.139:0/1030361805 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72a8013410 con 0x7f72b40f7920 2021-11-20T06:05:59.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.103+0000 7f72b900c700 1 -- 172.21.15.139:0/1030361805 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b410c4b0 con 0x7f72b40f7920 2021-11-20T06:05:59.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.103+0000 7f729b7fe700 1 -- 172.21.15.139:0/1030361805 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72a801c3c0 con 0x7f72b40f7920 2021-11-20T06:05:59.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.103+0000 7f72b900c700 1 -- 172.21.15.139:0/1030361805 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b410ca70 con 0x7f72b40f7920 2021-11-20T06:05:59.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.104+0000 7f72b900c700 1 -- 172.21.15.139:0/1030361805 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b4027a00 con 0x7f72b40f7920 2021-11-20T06:05:59.096 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.104+0000 7f729b7fe700 1 -- 172.21.15.139:0/1030361805 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f72a801d420 con 0x7f72b40f7920 2021-11-20T06:05:59.097 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.105+0000 7f729b7fe700 1 --2- 172.21.15.139:0/1030361805 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f729c05c2a0 0x7f729c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:05:59.097 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.105+0000 7f729b7fe700 1 -- 172.21.15.139:0/1030361805 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f72a8077df0 con 0x7f72b40f7920 2021-11-20T06:05:59.097 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.105+0000 7f72b2d9d700 1 --2- 172.21.15.139:0/1030361805 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f729c05c2a0 0x7f729c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:05:59.097 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.105+0000 7f72b2d9d700 1 --2- 172.21.15.139:0/1030361805 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f729c05c2a0 0x7f729c05e760 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f72b40f7780 tx=0x7f72a400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:05:59.098 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.109+0000 7f729b7fe700 1 -- 172.21.15.139:0/1030361805 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72a804a800 con 0x7f72b40f7920 2021-11-20T06:05:59.285 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.300+0000 7f72b900c700 1 -- 172.21.15.139:0/1030361805 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72b40fee10 con 0x7f72b40f7920 2021-11-20T06:05:59.286 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.301+0000 7f729b7fe700 1 -- 172.21.15.139:0/1030361805 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2943 (secure 0 0 0) 0x7f72a804de20 con 0x7f72b40f7920 2021-11-20T06:05:59.286 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:05:59.287 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":52},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":51,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:05:50.315512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14560":{"gid":14560,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.139:6829/1072553578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1072553578},{"type":"v1","addr":"172.21.15.139:6829","nonce":1072553578}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:05:59.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.302+0000 7f72b900c700 1 -- 172.21.15.139:0/1030361805 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f729c05c2a0 msgr2=0x7f729c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:59.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.302+0000 7f72b900c700 1 --2- 172.21.15.139:0/1030361805 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f729c05c2a0 0x7f729c05e760 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f72b40f7780 tx=0x7f72a400b040).stop 2021-11-20T06:05:59.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.302+0000 7f72b900c700 1 -- 172.21.15.139:0/1030361805 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72b40f7920 msgr2=0x7f72b4107380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:05:59.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.302+0000 7f72b900c700 1 --2- 172.21.15.139:0/1030361805 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72b40f7920 0x7f72b4107380 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f72a8009f50 tx=0x7f72a8009b30).stop 2021-11-20T06:05:59.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.302+0000 7f72b900c700 1 -- 172.21.15.139:0/1030361805 shutdown_connections 2021-11-20T06:05:59.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.302+0000 7f72b900c700 1 --2- 172.21.15.139:0/1030361805 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f729c05c2a0 0x7f729c05e760 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:59.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.302+0000 7f72b900c700 1 --2- 172.21.15.139:0/1030361805 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72b40f6680 0x7f72b4106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:59.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.302+0000 7f72b900c700 1 --2- 172.21.15.139:0/1030361805 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f72b40f7920 0x7f72b4107380 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:05:59.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.302+0000 7f72b900c700 1 -- 172.21.15.139:0/1030361805 >> 172.21.15.139:0/1030361805 conn(0x7f72b40f1de0 msgr2=0x7f72b40fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:05:59.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.302+0000 7f72b900c700 1 -- 172.21.15.139:0/1030361805 shutdown_connections 2021-11-20T06:05:59.291 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:05:59.302+0000 7f72b900c700 1 -- 172.21.15.139:0/1030361805 wait complete. 2021-11-20T06:05:59.291 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 52 2021-11-20T06:05:59.300 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:05:50.315512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14560': {'gid': 14560, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 49, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.139:6829/1072553578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 1072553578}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 1072553578}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:05:59.300 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14560:up:rejoin 2021-11-20T06:05:59.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:59 smithi178 conmon[36600]: cluster 2021-11-20T06:05:58.250412+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:05:59.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:59 smithi178 conmon[36600]: 265 : cluster [DBG] pgmap v234: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:05:59.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:05:59 smithi178 conmon[36600]: audit 2021-11-20T06:05:59.301420+0000 mon.smithi139 (mon.0) 673 : audit [DBG] from='client.? 172.21.15.139:0/1030361805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:05:59.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:59 smithi139 conmon[33501]: cluster 2021-11-20T06:05:58.250412+0000 mgr.smithi139.dplyxn (mgr.14162) 265 2021-11-20T06:05:59.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:59 smithi139 conmon[33501]: : cluster [DBG] pgmap v234: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:05:59.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:05:59 smithi139 conmon[33501]: audit 2021-11-20T06:05:59.301420+0000 mon.smithi139 (mon.0) 673 : audit [DBG] from='client.? 172.21.15.139:0/1030361805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:00.302 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:00.432 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.444+0000 7f199560c700 1 -- 172.21.15.139:0/1041740334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19900f65a0 msgr2=0x7f19900f69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:00.432 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.444+0000 7f199560c700 1 --2- 172.21.15.139:0/1041740334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19900f65a0 0x7f19900f69c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f1980004660 tx=0x7f1980009b30).stop 2021-11-20T06:06:00.433 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.444+0000 7f199560c700 1 -- 172.21.15.139:0/1041740334 shutdown_connections 2021-11-20T06:06:00.433 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.444+0000 7f199560c700 1 --2- 172.21.15.139:0/1041740334 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19900f7840 0x7f19900f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:00.434 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.444+0000 7f199560c700 1 --2- 172.21.15.139:0/1041740334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19900f65a0 0x7f19900f69c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:00.434 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.444+0000 7f199560c700 1 -- 172.21.15.139:0/1041740334 >> 172.21.15.139:0/1041740334 conn(0x7f19900f1ce0 msgr2=0x7f19900f4120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:00.434 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.444+0000 7f199560c700 1 -- 172.21.15.139:0/1041740334 shutdown_connections 2021-11-20T06:06:00.434 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.444+0000 7f199560c700 1 -- 172.21.15.139:0/1041740334 wait complete. 2021-11-20T06:06:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.445+0000 7f199560c700 1 Processor -- start 2021-11-20T06:06:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.445+0000 7f199560c700 1 -- start start 2021-11-20T06:06:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.445+0000 7f199560c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19900f65a0 0x7f1990106dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.445+0000 7f199560c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19900f7840 0x7f19901072b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:00.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.445+0000 7f199560c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19901078b0 con 0x7f19900f65a0 2021-11-20T06:06:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.445+0000 7f199560c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19901080f0 con 0x7f19900f7840 2021-11-20T06:06:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.445+0000 7f198ffff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19900f65a0 0x7f1990106dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.445+0000 7f198ffff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19900f65a0 0x7f1990106dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58020/0 (socket says 172.21.15.139:58020) 2021-11-20T06:06:00.436 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.445+0000 7f198ffff700 1 -- 172.21.15.139:0/963824083 learned_addr learned my addr 172.21.15.139:0/963824083 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:00.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.445+0000 7f198f7fe700 1 --2- 172.21.15.139:0/963824083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19900f7840 0x7f19901072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:00.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.446+0000 7f198ffff700 1 -- 172.21.15.139:0/963824083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19900f7840 msgr2=0x7f19901072b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:00.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.446+0000 7f198ffff700 1 --2- 172.21.15.139:0/963824083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19900f7840 0x7f19901072b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:00.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.446+0000 7f198ffff700 1 -- 172.21.15.139:0/963824083 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1980005040 con 0x7f19900f65a0 2021-11-20T06:06:00.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.446+0000 7f198ffff700 1 --2- 172.21.15.139:0/963824083 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19900f65a0 0x7f1990106dc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f1980004e30 tx=0x7f1980009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:00.438 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.446+0000 7f198cff9700 1 -- 172.21.15.139:0/963824083 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1980018070 con 0x7f19900f65a0 2021-11-20T06:06:00.438 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.446+0000 7f198cff9700 1 -- 172.21.15.139:0/963824083 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1980013410 con 0x7f19900f65a0 2021-11-20T06:06:00.438 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.446+0000 7f199560c700 1 -- 172.21.15.139:0/963824083 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1990108370 con 0x7f19900f65a0 2021-11-20T06:06:00.438 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.446+0000 7f198cff9700 1 -- 172.21.15.139:0/963824083 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f198001c3c0 con 0x7f19900f65a0 2021-11-20T06:06:00.439 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.446+0000 7f199560c700 1 -- 172.21.15.139:0/963824083 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1990108850 con 0x7f19900f65a0 2021-11-20T06:06:00.439 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.447+0000 7f199560c700 1 -- 172.21.15.139:0/963824083 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19900f5990 con 0x7f19900f65a0 2021-11-20T06:06:00.439 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.447+0000 7f198cff9700 1 -- 172.21.15.139:0/963824083 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f198001d420 con 0x7f19900f65a0 2021-11-20T06:06:00.439 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.447+0000 7f198cff9700 1 --2- 172.21.15.139:0/963824083 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f197805c300 0x7f197805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:00.439 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.447+0000 7f198cff9700 1 -- 172.21.15.139:0/963824083 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f1980077d70 con 0x7f19900f65a0 2021-11-20T06:06:00.440 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.448+0000 7f198f7fe700 1 --2- 172.21.15.139:0/963824083 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f197805c300 0x7f197805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:00.440 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.448+0000 7f198f7fe700 1 --2- 172.21.15.139:0/963824083 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f197805c300 0x7f197805e7c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1990106b50 tx=0x7f1984009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:00.441 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.451+0000 7f198cff9700 1 -- 172.21.15.139:0/963824083 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f198004a780 con 0x7f19900f65a0 2021-11-20T06:06:00.620 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.635+0000 7f199560c700 1 -- 172.21.15.139:0/963824083 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1990027b20 con 0x7f19900f65a0 2021-11-20T06:06:00.622 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.636+0000 7f198cff9700 1 -- 172.21.15.139:0/963824083 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2943 (secure 0 0 0) 0x7f198004dda0 con 0x7f19900f65a0 2021-11-20T06:06:00.622 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:00.623 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":52},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":51,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:05:50.315512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14560":{"gid":14560,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.139:6829/1072553578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1072553578},{"type":"v1","addr":"172.21.15.139:6829","nonce":1072553578}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:00.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.637+0000 7f199560c700 1 -- 172.21.15.139:0/963824083 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f197805c300 msgr2=0x7f197805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:00.626 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.637+0000 7f199560c700 1 --2- 172.21.15.139:0/963824083 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f197805c300 0x7f197805e7c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1990106b50 tx=0x7f1984009be0).stop 2021-11-20T06:06:00.626 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.637+0000 7f199560c700 1 -- 172.21.15.139:0/963824083 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19900f65a0 msgr2=0x7f1990106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:00.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.637+0000 7f199560c700 1 --2- 172.21.15.139:0/963824083 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19900f65a0 0x7f1990106dc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f1980004e30 tx=0x7f1980009b30).stop 2021-11-20T06:06:00.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.637+0000 7f199560c700 1 -- 172.21.15.139:0/963824083 shutdown_connections 2021-11-20T06:06:00.628 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.637+0000 7f199560c700 1 --2- 172.21.15.139:0/963824083 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f197805c300 0x7f197805e7c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:00.628 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.637+0000 7f199560c700 1 --2- 172.21.15.139:0/963824083 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f19900f65a0 0x7f1990106dc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:00.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.637+0000 7f199560c700 1 --2- 172.21.15.139:0/963824083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19900f7840 0x7f19901072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:00.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.637+0000 7f199560c700 1 -- 172.21.15.139:0/963824083 >> 172.21.15.139:0/963824083 conn(0x7f19900f1ce0 msgr2=0x7f19900fb290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:00.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.637+0000 7f199560c700 1 -- 172.21.15.139:0/963824083 shutdown_connections 2021-11-20T06:06:00.630 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:00.637+0000 7f199560c700 1 -- 172.21.15.139:0/963824083 wait complete. 2021-11-20T06:06:00.630 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 52 2021-11-20T06:06:00.635 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:05:50.315512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14560': {'gid': 14560, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 49, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.139:6829/1072553578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 1072553578}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 1072553578}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:00.636 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14560:up:rejoin 2021-11-20T06:06:01.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:01 smithi139 conmon[33501]: cluster 2021-11-20T06:06:00.251065+0000 mgr.smithi139.dplyxn (mgr.14162) 266 : cluster [DBG] pgmap v235: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:06:01.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:01 smithi139 conmon[33501]: 2021-11-20T06:06:01.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:01 smithi139 conmon[33501]: audit 2021-11-20T06:06:00.636472+0000 mon.smithi139 (mon.0) 674 : audit [DBG] from='client.? 172.21.15.139:0/963824083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:01.637 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:01.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:01 smithi178 conmon[36600]: cluster 2021-11-20T06:06:00.251065+0000 mgr.smithi139.dplyxn (mgr.14162) 266 : 2021-11-20T06:06:01.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:01 smithi178 conmon[36600]: cluster [DBG] pgmap v235: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:06:01.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:01 smithi178 conmon[36600]: audit 2021-11-20T06:06:00.636472+0000 mon.smithi139 (mon.0) 674 : audit [DBG] from='client.? 172.21.15.139:0/963824083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:01.756 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.769+0000 7fe51aba0700 1 -- 172.21.15.139:0/1807045055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5140f5580 msgr2=0x7fe5140f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:01.756 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.769+0000 7fe51aba0700 1 --2- 172.21.15.139:0/1807045055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5140f5580 0x7fe5140f59a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fe508004660 tx=0x7fe508009b30).stop 2021-11-20T06:06:01.756 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.769+0000 7fe51aba0700 1 -- 172.21.15.139:0/1807045055 shutdown_connections 2021-11-20T06:06:01.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.769+0000 7fe51aba0700 1 --2- 172.21.15.139:0/1807045055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5140f6820 0x7fe5140f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:01.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.769+0000 7fe51aba0700 1 --2- 172.21.15.139:0/1807045055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5140f5580 0x7fe5140f59a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:01.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.769+0000 7fe51aba0700 1 -- 172.21.15.139:0/1807045055 >> 172.21.15.139:0/1807045055 conn(0x7fe5140f0ce0 msgr2=0x7fe5140f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:01.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.769+0000 7fe51aba0700 1 -- 172.21.15.139:0/1807045055 shutdown_connections 2021-11-20T06:06:01.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.769+0000 7fe51aba0700 1 -- 172.21.15.139:0/1807045055 wait complete. 2021-11-20T06:06:01.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.769+0000 7fe51aba0700 1 Processor -- start 2021-11-20T06:06:01.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.769+0000 7fe51aba0700 1 -- start start 2021-11-20T06:06:01.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe51aba0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5140f5580 0x7fe5140604d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:01.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe51aba0700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5140f6820 0x7fe51410e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:01.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe51aba0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe514107a70 con 0x7fe5140f5580 2021-11-20T06:06:01.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe51aba0700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51410e580 con 0x7fe5140f6820 2021-11-20T06:06:01.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe51893c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5140f5580 0x7fe5140604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:01.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe51893c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5140f5580 0x7fe5140604d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58038/0 (socket says 172.21.15.139:58038) 2021-11-20T06:06:01.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe51893c700 1 -- 172.21.15.139:0/4233188017 learned_addr learned my addr 172.21.15.139:0/4233188017 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:01.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe51893c700 1 -- 172.21.15.139:0/4233188017 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5140f6820 msgr2=0x7fe51410e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:01.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe513fff700 1 --2- 172.21.15.139:0/4233188017 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5140f6820 0x7fe51410e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:01.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe51893c700 1 --2- 172.21.15.139:0/4233188017 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5140f6820 0x7fe51410e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:01.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe51893c700 1 -- 172.21.15.139:0/4233188017 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe508005040 con 0x7fe5140f5580 2021-11-20T06:06:01.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe513fff700 1 --2- 172.21.15.139:0/4233188017 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5140f6820 0x7fe51410e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:06:01.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe51893c700 1 --2- 172.21.15.139:0/4233188017 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5140f5580 0x7fe5140604d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fe508009f50 tx=0x7fe508009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:01.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe5117fa700 1 -- 172.21.15.139:0/4233188017 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe508018070 con 0x7fe5140f5580 2021-11-20T06:06:01.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe51aba0700 1 -- 172.21.15.139:0/4233188017 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe51410e780 con 0x7fe5140f5580 2021-11-20T06:06:01.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe5117fa700 1 -- 172.21.15.139:0/4233188017 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe508013410 con 0x7fe5140f5580 2021-11-20T06:06:01.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe5117fa700 1 -- 172.21.15.139:0/4233188017 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe50801c620 con 0x7fe5140f5580 2021-11-20T06:06:01.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.770+0000 7fe51aba0700 1 -- 172.21.15.139:0/4233188017 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe51410ecb0 con 0x7fe5140f5580 2021-11-20T06:06:01.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.771+0000 7fe51aba0700 1 -- 172.21.15.139:0/4233188017 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe514046330 con 0x7fe5140f5580 2021-11-20T06:06:01.764 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.772+0000 7fe5117fa700 1 -- 172.21.15.139:0/4233188017 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe50801d420 con 0x7fe5140f5580 2021-11-20T06:06:01.764 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.772+0000 7fe5117fa700 1 --2- 172.21.15.139:0/4233188017 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe50405c2a0 0x7fe50405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:01.764 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.772+0000 7fe5117fa700 1 -- 172.21.15.139:0/4233188017 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fe508077ff0 con 0x7fe5140f5580 2021-11-20T06:06:01.764 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.772+0000 7fe513fff700 1 --2- 172.21.15.139:0/4233188017 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe50405c2a0 0x7fe50405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:01.765 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.772+0000 7fe513fff700 1 --2- 172.21.15.139:0/4233188017 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe50405c2a0 0x7fe50405e760 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe50000a7e0 tx=0x7fe500008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:01.765 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.776+0000 7fe5117fa700 1 -- 172.21.15.139:0/4233188017 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe50804aa00 con 0x7fe5140f5580 2021-11-20T06:06:01.943 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.958+0000 7fe51aba0700 1 -- 172.21.15.139:0/4233188017 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe514027a00 con 0x7fe5140f5580 2021-11-20T06:06:01.944 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.959+0000 7fe5117fa700 1 -- 172.21.15.139:0/4233188017 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2943 (secure 0 0 0) 0x7fe50804e020 con 0x7fe5140f5580 2021-11-20T06:06:01.944 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:01.945 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":52},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":51,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:05:50.315512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14560":{"gid":14560,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.139:6829/1072553578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1072553578},{"type":"v1","addr":"172.21.15.139:6829","nonce":1072553578}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:01.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.960+0000 7fe51aba0700 1 -- 172.21.15.139:0/4233188017 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe50405c2a0 msgr2=0x7fe50405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:01.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.960+0000 7fe51aba0700 1 --2- 172.21.15.139:0/4233188017 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe50405c2a0 0x7fe50405e760 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe50000a7e0 tx=0x7fe500008040).stop 2021-11-20T06:06:01.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.960+0000 7fe51aba0700 1 -- 172.21.15.139:0/4233188017 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5140f5580 msgr2=0x7fe5140604d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:01.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.960+0000 7fe51aba0700 1 --2- 172.21.15.139:0/4233188017 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5140f5580 0x7fe5140604d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fe508009f50 tx=0x7fe508009b30).stop 2021-11-20T06:06:01.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.961+0000 7fe51aba0700 1 -- 172.21.15.139:0/4233188017 shutdown_connections 2021-11-20T06:06:01.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.961+0000 7fe51aba0700 1 --2- 172.21.15.139:0/4233188017 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe50405c2a0 0x7fe50405e760 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:01.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.961+0000 7fe51aba0700 1 --2- 172.21.15.139:0/4233188017 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe5140f5580 0x7fe5140604d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:01.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.961+0000 7fe51aba0700 1 --2- 172.21.15.139:0/4233188017 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5140f6820 0x7fe51410e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:01.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.961+0000 7fe51aba0700 1 -- 172.21.15.139:0/4233188017 >> 172.21.15.139:0/4233188017 conn(0x7fe5140f0ce0 msgr2=0x7fe5140fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:01.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.961+0000 7fe51aba0700 1 -- 172.21.15.139:0/4233188017 shutdown_connections 2021-11-20T06:06:01.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:01.961+0000 7fe51aba0700 1 -- 172.21.15.139:0/4233188017 wait complete. 2021-11-20T06:06:01.949 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 52 2021-11-20T06:06:01.957 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:05:50.315512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14560': {'gid': 14560, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 49, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.139:6829/1072553578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 1072553578}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 1072553578}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:01.958 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14560:up:rejoin 2021-11-20T06:06:02.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:02 smithi178 conmon[36600]: audit 2021-11-20T06:06:01.959726+0000 mon.smithi139 (mon.0) 675 : audit [DBG] from='client.? 172.21.15.139:0/4233188017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:02.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:02 smithi139 conmon[33501]: audit 2021-11-20T06:06:01.959726+0000 mon.smithi139 (mon.0) 675 : audit [DBG] from='client.? 172.21.15.139:0/4233188017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:02.959 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:03.089 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.102+0000 7fe0fe30a700 1 -- 172.21.15.139:0/819956462 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0f80f97e0 msgr2=0x7fe0f80fbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:03.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.102+0000 7fe0fe30a700 1 --2- 172.21.15.139:0/819956462 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0f80f97e0 0x7fe0f80fbc30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fe0e0004660 tx=0x7fe0e0009b30).stop 2021-11-20T06:06:03.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.102+0000 7fe0fe30a700 1 -- 172.21.15.139:0/819956462 shutdown_connections 2021-11-20T06:06:03.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.102+0000 7fe0fe30a700 1 --2- 172.21.15.139:0/819956462 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0f80f97e0 0x7fe0f80fbc30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:03.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.102+0000 7fe0fe30a700 1 --2- 172.21.15.139:0/819956462 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0f8060730 0x7fe0f80f9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:03.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.102+0000 7fe0fe30a700 1 -- 172.21.15.139:0/819956462 >> 172.21.15.139:0/819956462 conn(0x7fe0f80f0ce0 msgr2=0x7fe0f80f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:03.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.102+0000 7fe0fe30a700 1 -- 172.21.15.139:0/819956462 shutdown_connections 2021-11-20T06:06:03.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.102+0000 7fe0fe30a700 1 -- 172.21.15.139:0/819956462 wait complete. 2021-11-20T06:06:03.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.103+0000 7fe0fe30a700 1 Processor -- start 2021-11-20T06:06:03.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.103+0000 7fe0fe30a700 1 -- start start 2021-11-20T06:06:03.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.103+0000 7fe0fe30a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0f8060730 0x7fe0f8106f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:03.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.103+0000 7fe0fe30a700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0f80f97e0 0x7fe0f81073f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:03.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.103+0000 7fe0fe30a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f81079a0 con 0x7fe0f8060730 2021-11-20T06:06:03.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.103+0000 7fe0fe30a700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f8107ae0 con 0x7fe0f80f97e0 2021-11-20T06:06:03.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.103+0000 7fe0f7fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0f8060730 0x7fe0f8106f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:03.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.103+0000 7fe0f7fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0f8060730 0x7fe0f8106f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58048/0 (socket says 172.21.15.139:58048) 2021-11-20T06:06:03.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.103+0000 7fe0f7fff700 1 -- 172.21.15.139:0/2457148386 learned_addr learned my addr 172.21.15.139:0/2457148386 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:03.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.103+0000 7fe0effff700 1 --2- 172.21.15.139:0/2457148386 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0f80f97e0 0x7fe0f81073f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:03.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.103+0000 7fe0f7fff700 1 -- 172.21.15.139:0/2457148386 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0f80f97e0 msgr2=0x7fe0f81073f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:03.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.103+0000 7fe0f7fff700 1 --2- 172.21.15.139:0/2457148386 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0f80f97e0 0x7fe0f81073f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:03.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.103+0000 7fe0f7fff700 1 -- 172.21.15.139:0/2457148386 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e0005040 con 0x7fe0f8060730 2021-11-20T06:06:03.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.103+0000 7fe0effff700 1 --2- 172.21.15.139:0/2457148386 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0f80f97e0 0x7fe0f81073f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:06:03.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.103+0000 7fe0f7fff700 1 --2- 172.21.15.139:0/2457148386 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0f8060730 0x7fe0f8106f00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fe0e80026d0 tx=0x7fe0e8009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:03.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.104+0000 7fe0f57fa700 1 -- 172.21.15.139:0/2457148386 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0e8009320 con 0x7fe0f8060730 2021-11-20T06:06:03.096 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.104+0000 7fe0fe30a700 1 -- 172.21.15.139:0/2457148386 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f810c580 con 0x7fe0f8060730 2021-11-20T06:06:03.096 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.104+0000 7fe0f57fa700 1 -- 172.21.15.139:0/2457148386 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0e800f040 con 0x7fe0f8060730 2021-11-20T06:06:03.096 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.104+0000 7fe0f57fa700 1 -- 172.21.15.139:0/2457148386 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0e80147e0 con 0x7fe0f8060730 2021-11-20T06:06:03.096 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.104+0000 7fe0fe30a700 1 -- 172.21.15.139:0/2457148386 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f810cb40 con 0x7fe0f8060730 2021-11-20T06:06:03.096 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.105+0000 7fe0fe30a700 1 -- 172.21.15.139:0/2457148386 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f8027a00 con 0x7fe0f8060730 2021-11-20T06:06:03.097 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.105+0000 7fe0f57fa700 1 -- 172.21.15.139:0/2457148386 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe0e8009480 con 0x7fe0f8060730 2021-11-20T06:06:03.098 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.105+0000 7fe0f57fa700 1 --2- 172.21.15.139:0/2457148386 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe0e405c2f0 0x7fe0e405e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:03.098 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.105+0000 7fe0f57fa700 1 -- 172.21.15.139:0/2457148386 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fe0e8076970 con 0x7fe0f8060730 2021-11-20T06:06:03.098 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.105+0000 7fe0effff700 1 --2- 172.21.15.139:0/2457148386 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe0e405c2f0 0x7fe0e405e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:03.098 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.106+0000 7fe0effff700 1 --2- 172.21.15.139:0/2457148386 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe0e405c2f0 0x7fe0e405e7b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fe0e0004630 tx=0x7fe0e0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:03.099 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.109+0000 7fe0f57fa700 1 -- 172.21.15.139:0/2457148386 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0e80492c0 con 0x7fe0f8060730 2021-11-20T06:06:03.278 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.294+0000 7fe0fe30a700 1 -- 172.21.15.139:0/2457148386 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0f80fee10 con 0x7fe0f8060730 2021-11-20T06:06:03.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.294+0000 7fe0f57fa700 1 -- 172.21.15.139:0/2457148386 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2943 (secure 0 0 0) 0x7fe0e804c8e0 con 0x7fe0f8060730 2021-11-20T06:06:03.280 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:03.280 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":52},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":51,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:05:50.315512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14560":{"gid":14560,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.139:6829/1072553578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1072553578},{"type":"v1","addr":"172.21.15.139:6829","nonce":1072553578}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:03.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.296+0000 7fe0fe30a700 1 -- 172.21.15.139:0/2457148386 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe0e405c2f0 msgr2=0x7fe0e405e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:03.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.296+0000 7fe0fe30a700 1 --2- 172.21.15.139:0/2457148386 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe0e405c2f0 0x7fe0e405e7b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fe0e0004630 tx=0x7fe0e0009b30).stop 2021-11-20T06:06:03.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.296+0000 7fe0fe30a700 1 -- 172.21.15.139:0/2457148386 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0f8060730 msgr2=0x7fe0f8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:03.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.296+0000 7fe0fe30a700 1 --2- 172.21.15.139:0/2457148386 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0f8060730 0x7fe0f8106f00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fe0e80026d0 tx=0x7fe0e8009de0).stop 2021-11-20T06:06:03.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.296+0000 7fe0fe30a700 1 -- 172.21.15.139:0/2457148386 shutdown_connections 2021-11-20T06:06:03.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.296+0000 7fe0fe30a700 1 --2- 172.21.15.139:0/2457148386 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe0e405c2f0 0x7fe0e405e7b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:03.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.296+0000 7fe0fe30a700 1 --2- 172.21.15.139:0/2457148386 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe0f8060730 0x7fe0f8106f00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:03.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.296+0000 7fe0fe30a700 1 --2- 172.21.15.139:0/2457148386 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0f80f97e0 0x7fe0f81073f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:03.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.296+0000 7fe0fe30a700 1 -- 172.21.15.139:0/2457148386 >> 172.21.15.139:0/2457148386 conn(0x7fe0f80f0ce0 msgr2=0x7fe0f80f30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:03.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.296+0000 7fe0fe30a700 1 -- 172.21.15.139:0/2457148386 shutdown_connections 2021-11-20T06:06:03.284 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:03.296+0000 7fe0fe30a700 1 -- 172.21.15.139:0/2457148386 wait complete. 2021-11-20T06:06:03.284 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 52 2021-11-20T06:06:03.293 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:05:50.315512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14560': {'gid': 14560, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 49, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.139:6829/1072553578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 1072553578}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 1072553578}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:03.293 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14560:up:rejoin 2021-11-20T06:06:03.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:03 smithi178 conmon[36600]: cluster 2021-11-20T06:06:02.251482+0000 mgr.smithi139.dplyxn (mgr.14162) 267 : cluster [DBG] pgmap v236: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T06:06:03.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:03 smithi178 conmon[36600]: audit 2021-11-20T06:06:03.295217+0000 2021-11-20T06:06:03.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:03 smithi178 conmon[36600]: mon.smithi139 (mon.0) 676 : audit [DBG] from='client.? 172.21.15.139:0/2457148386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:03.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:03 smithi139 conmon[33501]: cluster 2021-11-20T06:06:02.251482+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:06:03.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:03 smithi139 conmon[33501]: 267 : cluster [DBG] pgmap v236: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T06:06:03.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:03 smithi139 conmon[33501]: audit 2021-11-20T06:06:03.295217+0000 mon.smithi139 (mon.0) 676 : audit [DBG] from='client.? 172.21.15.139:0/2457148386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:04.296 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:04.430 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.439+0000 7f61f1ac5700 1 -- 172.21.15.139:0/800919002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61ec0f5300 msgr2=0x7f61ec0f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:04.431 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.439+0000 7f61f1ac5700 1 --2- 172.21.15.139:0/800919002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61ec0f5300 0x7f61ec0f7a80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f61e0004660 tx=0x7f61e0009b30).stop 2021-11-20T06:06:04.431 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.439+0000 7f61f1ac5700 1 -- 172.21.15.139:0/800919002 shutdown_connections 2021-11-20T06:06:04.432 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.439+0000 7f61f1ac5700 1 --2- 172.21.15.139:0/800919002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61ec0f5300 0x7f61ec0f7a80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:04.432 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.439+0000 7f61f1ac5700 1 --2- 172.21.15.139:0/800919002 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61ec0f4960 0x7f61ec0f4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:04.433 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.439+0000 7f61f1ac5700 1 -- 172.21.15.139:0/800919002 >> 172.21.15.139:0/800919002 conn(0x7f61ec0f0440 msgr2=0x7f61ec0f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:04.433 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.439+0000 7f61f1ac5700 1 -- 172.21.15.139:0/800919002 shutdown_connections 2021-11-20T06:06:04.433 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.439+0000 7f61f1ac5700 1 -- 172.21.15.139:0/800919002 wait complete. 2021-11-20T06:06:04.434 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.439+0000 7f61f1ac5700 1 Processor -- start 2021-11-20T06:06:04.434 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.439+0000 7f61f1ac5700 1 -- start start 2021-11-20T06:06:04.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61f1ac5700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61ec0f4960 0x7f61ec106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:04.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61f1ac5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61ec0f5300 0x7f61ec1073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:04.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61f1ac5700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec107970 con 0x7f61ec0f5300 2021-11-20T06:06:04.436 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61f1ac5700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec107ab0 con 0x7f61ec0f4960 2021-11-20T06:06:04.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61eaffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61ec0f5300 0x7f61ec1073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:04.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61eaffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61ec0f5300 0x7f61ec1073c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58060/0 (socket says 172.21.15.139:58060) 2021-11-20T06:06:04.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61eaffd700 1 -- 172.21.15.139:0/1627549000 learned_addr learned my addr 172.21.15.139:0/1627549000 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:04.438 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61eb7fe700 1 --2- 172.21.15.139:0/1627549000 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61ec0f4960 0x7f61ec106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:04.438 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61eaffd700 1 -- 172.21.15.139:0/1627549000 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61ec0f4960 msgr2=0x7f61ec106ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:04.439 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61eaffd700 1 --2- 172.21.15.139:0/1627549000 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61ec0f4960 0x7f61ec106ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:04.439 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61eaffd700 1 -- 172.21.15.139:0/1627549000 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e0005040 con 0x7f61ec0f5300 2021-11-20T06:06:04.440 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61eaffd700 1 --2- 172.21.15.139:0/1627549000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61ec0f5300 0x7f61ec1073c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f61e0004e30 tx=0x7f61e0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:04.440 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61d3fff700 1 -- 172.21.15.139:0/1627549000 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61e0018070 con 0x7f61ec0f5300 2021-11-20T06:06:04.440 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61d3fff700 1 -- 172.21.15.139:0/1627549000 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61e0013410 con 0x7f61ec0f5300 2021-11-20T06:06:04.441 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61f1ac5700 1 -- 172.21.15.139:0/1627549000 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61ec10c4f0 con 0x7f61ec0f5300 2021-11-20T06:06:04.441 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61d3fff700 1 -- 172.21.15.139:0/1627549000 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61e001de20 con 0x7f61ec0f5300 2021-11-20T06:06:04.442 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.440+0000 7f61f1ac5700 1 -- 172.21.15.139:0/1627549000 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ec10cab0 con 0x7f61ec0f5300 2021-11-20T06:06:04.442 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.441+0000 7f61f1ac5700 1 -- 172.21.15.139:0/1627549000 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61ec027a00 con 0x7f61ec0f5300 2021-11-20T06:06:04.442 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.442+0000 7f61d3fff700 1 -- 172.21.15.139:0/1627549000 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f61e001d420 con 0x7f61ec0f5300 2021-11-20T06:06:04.443 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.442+0000 7f61d3fff700 1 --2- 172.21.15.139:0/1627549000 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f61d405c2a0 0x7f61d405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:04.444 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.442+0000 7f61d3fff700 1 -- 172.21.15.139:0/1627549000 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f61e0077d20 con 0x7f61ec0f5300 2021-11-20T06:06:04.444 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.442+0000 7f61eb7fe700 1 --2- 172.21.15.139:0/1627549000 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f61d405c2a0 0x7f61d405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:04.444 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.442+0000 7f61eb7fe700 1 --2- 172.21.15.139:0/1627549000 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f61d405c2a0 0x7f61d405e760 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f61ec05f7e0 tx=0x7f61dc00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:04.445 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.446+0000 7f61d3fff700 1 -- 172.21.15.139:0/1627549000 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61e004a7b0 con 0x7f61ec0f5300 2021-11-20T06:06:04.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.631+0000 7f61f1ac5700 1 -- 172.21.15.139:0/1627549000 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61ec0fee50 con 0x7f61ec0f5300 2021-11-20T06:06:04.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.631+0000 7f61d3fff700 1 -- 172.21.15.139:0/1627549000 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2943 (secure 0 0 0) 0x7f61e004ddd0 con 0x7f61ec0f5300 2021-11-20T06:06:04.617 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:04.617 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":52},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":51,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:05:50.315512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14560":{"gid":14560,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":49,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.139:6829/1072553578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":1072553578},{"type":"v1","addr":"172.21.15.139:6829","nonce":1072553578}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:04.618 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.633+0000 7f61f1ac5700 1 -- 172.21.15.139:0/1627549000 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f61d405c2a0 msgr2=0x7f61d405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:04.619 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.633+0000 7f61f1ac5700 1 --2- 172.21.15.139:0/1627549000 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f61d405c2a0 0x7f61d405e760 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f61ec05f7e0 tx=0x7f61dc00b040).stop 2021-11-20T06:06:04.619 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.633+0000 7f61f1ac5700 1 -- 172.21.15.139:0/1627549000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61ec0f5300 msgr2=0x7f61ec1073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:04.619 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.633+0000 7f61f1ac5700 1 --2- 172.21.15.139:0/1627549000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61ec0f5300 0x7f61ec1073c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f61e0004e30 tx=0x7f61e0009b30).stop 2021-11-20T06:06:04.619 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.633+0000 7f61f1ac5700 1 -- 172.21.15.139:0/1627549000 shutdown_connections 2021-11-20T06:06:04.620 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.633+0000 7f61f1ac5700 1 --2- 172.21.15.139:0/1627549000 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f61d405c2a0 0x7f61d405e760 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:04.620 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.633+0000 7f61f1ac5700 1 --2- 172.21.15.139:0/1627549000 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61ec0f4960 0x7f61ec106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:04.620 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.633+0000 7f61f1ac5700 1 --2- 172.21.15.139:0/1627549000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f61ec0f5300 0x7f61ec1073c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:04.621 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.633+0000 7f61f1ac5700 1 -- 172.21.15.139:0/1627549000 >> 172.21.15.139:0/1627549000 conn(0x7f61ec0f0440 msgr2=0x7f61ec0fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:04.621 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.633+0000 7f61f1ac5700 1 -- 172.21.15.139:0/1627549000 shutdown_connections 2021-11-20T06:06:04.621 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:04.633+0000 7f61f1ac5700 1 -- 172.21.15.139:0/1627549000 wait complete. 2021-11-20T06:06:04.621 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 52 2021-11-20T06:06:04.630 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:05:50.315512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14560': {'gid': 14560, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 49, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.139:6829/1072553578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 1072553578}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 1072553578}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:04.631 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14560:up:rejoin 2021-11-20T06:06:05.632 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:05.658 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:05 smithi139 conmon[33501]: debug 2021-11-20T06:06:05.345+0000 7f2a21654700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:06:05.658 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:05 smithi139 conmon[33501]: cluster 2021-11-20T06:06:04.251936+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:06:05.659 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:05 smithi139 conmon[33501]: 14162) 268 : cluster [DBG] pgmap v237: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:06:05.659 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:05 smithi139 conmon[33501]: audit 2021-11-20T06:06:04.632065+0000 mon.smithi139 (mon.0) 677 : audit [DBG] from='client.? 172.21.15.139:0/1627549000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:05.659 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:05 smithi139 conmon[33501]: cluster 2021-11-20T06:06:05.659 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:05 smithi139 conmon[33501]: 2021-11-20T06:06:05.301246+0000 mon.smithi139 (mon.0) 678 : cluster [INF] daemon mds.cephfs.smithi139.oudezh restarted 2021-11-20T06:06:05.659 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:05 smithi139 conmon[33501]: cluster 2021-11 2021-11-20T06:06:05.660 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:05 smithi139 conmon[33501]: -20T06:06:05.308206+0000 mon.smithi139 (mon.0) 679 : cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2021-11-20T06:06:05.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:05 smithi178 conmon[36600]: cluster 2021-11-20T06:06:04.251936+0000 mgr.smithi139.dplyxn (mgr.14162) 268 : cluster [DBG] pgmap v237: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:06:05.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:05 smithi178 conmon[36600]: audit 2021-11-20T06:06:04.632065+0000 mon.smithi139 (mon.0) 677 : audit [DBG] from='client.? 172.21.15.139:0/1627549000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:05.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:05 smithi178 conmon[36600]: cluster 2021-11-20T06:06:05.301246+0000 mon.smithi139 (mon.0) 678 : cluster [INF] daemon mds.cephfs.smithi139.oudezh restarted 2021-11-20T06:06:05.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:05 smithi178 conmon[36600]: cluster 2021-11- 2021-11-20T06:06:05.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:05 smithi178 conmon[36600]: 20T06:06:05.308206+0000 2021-11-20T06:06:05.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:05 smithi178 conmon[36600]: mon.smithi139 (mon.0) 679 : cluster [DBG] 2021-11-20T06:06:05.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:05 smithi178 conmon[36600]: osdmap e50: 6 total, 6 up, 6 in 2021-11-20T06:06:05.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.769+0000 7fc7988f4700 1 -- 172.21.15.139:0/3315272298 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7940f6850 msgr2=0x7fc7940f6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:05.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.769+0000 7fc7988f4700 1 --2- 172.21.15.139:0/3315272298 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7940f6850 0x7fc7940f6cb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fc788004660 tx=0x7fc788009b30).stop 2021-11-20T06:06:05.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.769+0000 7fc7988f4700 1 -- 172.21.15.139:0/3315272298 shutdown_connections 2021-11-20T06:06:05.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.769+0000 7fc7988f4700 1 --2- 172.21.15.139:0/3315272298 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7940f6850 0x7fc7940f6cb0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:05.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.769+0000 7fc7988f4700 1 --2- 172.21.15.139:0/3315272298 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7940f55b0 0x7fc7940f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:05.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.769+0000 7fc7988f4700 1 -- 172.21.15.139:0/3315272298 >> 172.21.15.139:0/3315272298 conn(0x7fc7940f0d10 msgr2=0x7fc7940f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:05.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.769+0000 7fc7988f4700 1 -- 172.21.15.139:0/3315272298 shutdown_connections 2021-11-20T06:06:05.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.769+0000 7fc7988f4700 1 -- 172.21.15.139:0/3315272298 wait complete. 2021-11-20T06:06:05.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc7988f4700 1 Processor -- start 2021-11-20T06:06:05.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc7988f4700 1 -- start start 2021-11-20T06:06:05.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc7988f4700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7940f55b0 0x7fc794060570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:05.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc7988f4700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7940f6850 0x7fc79410bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:05.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc7988f4700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79410c1c0 con 0x7fc7940f55b0 2021-11-20T06:06:05.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc7988f4700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79410c300 con 0x7fc7940f6850 2021-11-20T06:06:05.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc79259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7940f55b0 0x7fc794060570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:05.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc79259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7940f55b0 0x7fc794060570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58080/0 (socket says 172.21.15.139:58080) 2021-11-20T06:06:05.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc79259c700 1 -- 172.21.15.139:0/3972234087 learned_addr learned my addr 172.21.15.139:0/3972234087 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:05.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc79259c700 1 -- 172.21.15.139:0/3972234087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7940f6850 msgr2=0x7fc79410bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:05.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc79259c700 1 --2- 172.21.15.139:0/3972234087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7940f6850 0x7fc79410bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:05.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc79259c700 1 -- 172.21.15.139:0/3972234087 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc788005040 con 0x7fc7940f55b0 2021-11-20T06:06:05.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc79259c700 1 --2- 172.21.15.139:0/3972234087 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7940f55b0 0x7fc794060570 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fc7840093f0 tx=0x7fc784006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:05.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc782ffd700 1 -- 172.21.15.139:0/3972234087 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc784009ea0 con 0x7fc7940f55b0 2021-11-20T06:06:05.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc782ffd700 1 -- 172.21.15.139:0/3972234087 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc784013af0 con 0x7fc7940f55b0 2021-11-20T06:06:05.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc7988f4700 1 -- 172.21.15.139:0/3972234087 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc79410c590 con 0x7fc7940f55b0 2021-11-20T06:06:05.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc782ffd700 1 -- 172.21.15.139:0/3972234087 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc78400dad0 con 0x7fc7940f55b0 2021-11-20T06:06:05.763 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.770+0000 7fc7988f4700 1 -- 172.21.15.139:0/3972234087 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79410cb50 con 0x7fc7940f55b0 2021-11-20T06:06:05.763 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.771+0000 7fc7988f4700 1 -- 172.21.15.139:0/3972234087 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc794027a00 con 0x7fc7940f55b0 2021-11-20T06:06:05.763 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.772+0000 7fc782ffd700 1 -- 172.21.15.139:0/3972234087 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc78401a060 con 0x7fc7940f55b0 2021-11-20T06:06:05.763 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.772+0000 7fc782ffd700 1 --2- 172.21.15.139:0/3972234087 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc77c05c2a0 0x7fc77c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:05.764 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.772+0000 7fc782ffd700 1 -- 172.21.15.139:0/3972234087 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fc7840761f0 con 0x7fc7940f55b0 2021-11-20T06:06:05.764 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.772+0000 7fc791d9b700 1 --2- 172.21.15.139:0/3972234087 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc77c05c2a0 0x7fc77c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:05.764 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.772+0000 7fc791d9b700 1 --2- 172.21.15.139:0/3972234087 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc77c05c2a0 0x7fc77c05e760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fc788000c00 tx=0x7fc788009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:05.765 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.776+0000 7fc782ffd700 1 -- 172.21.15.139:0/3972234087 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc784048bb0 con 0x7fc7940f55b0 2021-11-20T06:06:05.942 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.957+0000 7fc7988f4700 1 -- 172.21.15.139:0/3972234087 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7940fdde0 con 0x7fc7940f55b0 2021-11-20T06:06:05.943 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.958+0000 7fc782ffd700 1 -- 172.21.15.139:0/3972234087 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2944 (secure 0 0 0) 0x7fc78404c1d0 con 0x7fc7940f55b0 2021-11-20T06:06:05.943 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:05.944 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":53},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":54,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:05.351588+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14564":{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":54,"state":"up:replay","state_seq":2,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:05.945 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.960+0000 7fc7988f4700 1 -- 172.21.15.139:0/3972234087 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc77c05c2a0 msgr2=0x7fc77c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:05.945 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.960+0000 7fc7988f4700 1 --2- 172.21.15.139:0/3972234087 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc77c05c2a0 0x7fc77c05e760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fc788000c00 tx=0x7fc788009b30).stop 2021-11-20T06:06:05.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.960+0000 7fc7988f4700 1 -- 172.21.15.139:0/3972234087 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7940f55b0 msgr2=0x7fc794060570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:05.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.960+0000 7fc7988f4700 1 --2- 172.21.15.139:0/3972234087 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7940f55b0 0x7fc794060570 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fc7840093f0 tx=0x7fc784006ec0).stop 2021-11-20T06:06:05.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.960+0000 7fc7988f4700 1 -- 172.21.15.139:0/3972234087 shutdown_connections 2021-11-20T06:06:05.946 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.960+0000 7fc7988f4700 1 --2- 172.21.15.139:0/3972234087 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc77c05c2a0 0x7fc77c05e760 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:05.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.960+0000 7fc7988f4700 1 --2- 172.21.15.139:0/3972234087 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7940f55b0 0x7fc794060570 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:05.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.960+0000 7fc7988f4700 1 --2- 172.21.15.139:0/3972234087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7940f6850 0x7fc79410bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:05.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.960+0000 7fc7988f4700 1 -- 172.21.15.139:0/3972234087 >> 172.21.15.139:0/3972234087 conn(0x7fc7940f0d10 msgr2=0x7fc7940fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:05.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.960+0000 7fc7988f4700 1 -- 172.21.15.139:0/3972234087 shutdown_connections 2021-11-20T06:06:05.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:05.960+0000 7fc7988f4700 1 -- 172.21.15.139:0/3972234087 wait complete. 2021-11-20T06:06:05.948 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 54 2021-11-20T06:06:05.956 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:05.351588+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14564}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14564': {'gid': 14564, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 54, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.139:6827/3938771458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 3938771458}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 3938771458}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:05.957 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14564:up:replay 2021-11-20T06:06:06.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: cluster 2021-11-20T06:06:05.346517+0000 mon.smithi139 (mon.0) 680 2021-11-20T06:06:06.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:06:06.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: cluster 2021-11-20T06:06:05.351497+0000 mon.smithi139 (mon.0) 681 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2991819277,v1:172.21.15.139:6829/2991819277] up:boot 2021-11-20T06:06:06.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: cluster 2021-11-20T06:06:05.351548+0000 2021-11-20T06:06:06.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: mon.smithi139 (mon.0) 682 : cluster [INF] Standby daemon mds.cephfs.smithi139.riczqb assigned to filesystem cephfs as rank 0 2021-11-20T06:06:06.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: cluster 2021-11-20 2021-11-20T06:06:06.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: T06:06:05.351664+0000 mon.smithi139 (mon.0) 683 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:06:06.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: 2021-11-20T06:06:06.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: cluster 2021-11-20T06:06:06.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: 2021-11-20T06:06:05.351995+0000 mon.smithi139 ( 2021-11-20T06:06:06.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: mon.0) 684 : cluster [DBG] fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T06:06:06.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: audit 2021-11- 2021-11-20T06:06:06.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: 20T06:06:05.352118+0000 mon.smithi139 (mon. 2021-11-20T06:06:06.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: 0) 685 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.oudezh"}]: dispatch 2021-11-20T06:06:06.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: cluster 2021-11-20 2021-11-20T06:06:06.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: T06:06:05.355639+0000 mon.smithi139 (mon 2021-11-20T06:06:06.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: .0) 686 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:replay} 3 up:standby 2021-11-20T06:06:06.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: audit 2021- 2021-11-20T06:06:06.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: 11-20T06:06:05.958598+0000 mon.smithi139 2021-11-20T06:06:06.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:06 smithi178 conmon[36600]: (mon.0) 687 : audit [DBG] from='client.? 172.21.15.139:0/3972234087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:06.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: cluster 2021-11-20T06:06:05.346517+0000 mon.smithi139 (mon.0) 680 : 2021-11-20T06:06:06.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:06:06.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: cluster 2021-11-20T06:06:05.351497+0000 mon.smithi139 (mon.0) 681 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2991819277,v1:172.21.15.139:6829/2991819277] up:boot 2021-11-20T06:06:06.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: cluster 2021-11-20T06:06:05.351548+0000 mon.smithi139 (mon. 2021-11-20T06:06:06.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: 0) 682 : cluster [INF] Standby daemon mds.cephfs.smithi139.riczqb assigned to filesystem cephfs as rank 0 2021-11-20T06:06:06.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: cluster 2021-11-20T06:06:05.351664+0000 mon.smithi139 2021-11-20T06:06:06.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: (mon.0) 683 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:06:06.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: cluster 2021-11-20T06:06: 2021-11-20T06:06:06.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: 05.351995+0000 mon.smithi139 (mon.0) 684 : cluster [DBG] 2021-11-20T06:06:06.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: fsmap cephfs:0/1 4 up:standby, 1 failed 2021-11-20T06:06:06.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: audit 2021-11-20T06:06:05 2021-11-20T06:06:06.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: .352118+0000 mon.smithi139 (mon.0) 685 : audit [DBG] 2021-11-20T06:06:06.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.oudezh"}]: dispatch 2021-11-20T06:06:06.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: cluster 2021 2021-11-20T06:06:06.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: -11-20T06:06:05.355639+0000 mon.smithi139 ( 2021-11-20T06:06:06.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: mon.0) 686 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:replay} 3 up:standby 2021-11-20T06:06:06.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: audit 2021-11-20T06:06:06.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: 2021-11-20T06:06:05.958598+0000 mon.smithi139 2021-11-20T06:06:06.782 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:06 smithi139 conmon[33501]: (mon.0) 687 : audit [DBG] from='client.? 172.21.15.139:0/3972234087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:06.958 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:07.086 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.099+0000 7ff368fe3700 1 -- 172.21.15.139:0/3834415673 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3640f55b0 msgr2=0x7ff3640f59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:07.086 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.099+0000 7ff368fe3700 1 --2- 172.21.15.139:0/3834415673 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3640f55b0 0x7ff3640f59d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7ff3540099e0 tx=0x7ff354004d10).stop 2021-11-20T06:06:07.087 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.099+0000 7ff368fe3700 1 -- 172.21.15.139:0/3834415673 shutdown_connections 2021-11-20T06:06:07.087 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.099+0000 7ff368fe3700 1 --2- 172.21.15.139:0/3834415673 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3640f6850 0x7ff3640f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:07.087 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.099+0000 7ff368fe3700 1 --2- 172.21.15.139:0/3834415673 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3640f55b0 0x7ff3640f59d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:07.087 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.099+0000 7ff368fe3700 1 -- 172.21.15.139:0/3834415673 >> 172.21.15.139:0/3834415673 conn(0x7ff3640f0d10 msgr2=0x7ff3640f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:07.088 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.099+0000 7ff368fe3700 1 -- 172.21.15.139:0/3834415673 shutdown_connections 2021-11-20T06:06:07.088 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.099+0000 7ff368fe3700 1 -- 172.21.15.139:0/3834415673 wait complete. 2021-11-20T06:06:07.088 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.099+0000 7ff368fe3700 1 Processor -- start 2021-11-20T06:06:07.089 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.099+0000 7ff368fe3700 1 -- start start 2021-11-20T06:06:07.089 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff368fe3700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3640f55b0 0x7ff36405ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:07.089 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff368fe3700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3640f6850 0x7ff36410e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:07.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff368fe3700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff364107b70 con 0x7ff3640f6850 2021-11-20T06:06:07.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff368fe3700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36410e680 con 0x7ff3640f55b0 2021-11-20T06:06:07.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff361d9b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3640f6850 0x7ff36410e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:07.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff361d9b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3640f6850 0x7ff36410e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58094/0 (socket says 172.21.15.139:58094) 2021-11-20T06:06:07.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff361d9b700 1 -- 172.21.15.139:0/9510953 learned_addr learned my addr 172.21.15.139:0/9510953 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:07.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff36259c700 1 --2- 172.21.15.139:0/9510953 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3640f55b0 0x7ff36405ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:07.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff361d9b700 1 -- 172.21.15.139:0/9510953 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3640f55b0 msgr2=0x7ff36405ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:07.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff361d9b700 1 --2- 172.21.15.139:0/9510953 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3640f55b0 0x7ff36405ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:07.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff361d9b700 1 -- 172.21.15.139:0/9510953 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff354009640 con 0x7ff3640f6850 2021-11-20T06:06:07.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff36259c700 1 --2- 172.21.15.139:0/9510953 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3640f55b0 0x7ff36405ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:06:07.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff361d9b700 1 --2- 172.21.15.139:0/9510953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3640f6850 0x7ff36410e190 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7ff3580026d0 tx=0x7ff358009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:07.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff352ffd700 1 -- 172.21.15.139:0/9510953 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff358009320 con 0x7ff3640f6850 2021-11-20T06:06:07.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff368fe3700 1 -- 172.21.15.139:0/9510953 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36410e820 con 0x7ff3640f6850 2021-11-20T06:06:07.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff352ffd700 1 -- 172.21.15.139:0/9510953 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff35800f040 con 0x7ff3640f6850 2021-11-20T06:06:07.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff352ffd700 1 -- 172.21.15.139:0/9510953 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3580147e0 con 0x7ff3640f6850 2021-11-20T06:06:07.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.100+0000 7ff368fe3700 1 -- 172.21.15.139:0/9510953 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36410ed30 con 0x7ff3640f6850 2021-11-20T06:06:07.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.101+0000 7ff368fe3700 1 -- 172.21.15.139:0/9510953 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff364046330 con 0x7ff3640f6850 2021-11-20T06:06:07.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.102+0000 7ff352ffd700 1 -- 172.21.15.139:0/9510953 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff358009480 con 0x7ff3640f6850 2021-11-20T06:06:07.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.102+0000 7ff352ffd700 1 --2- 172.21.15.139:0/9510953 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff34c05c2a0 0x7ff34c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:07.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.102+0000 7ff352ffd700 1 -- 172.21.15.139:0/9510953 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7ff3580768b0 con 0x7ff3640f6850 2021-11-20T06:06:07.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.102+0000 7ff36259c700 1 --2- 172.21.15.139:0/9510953 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff34c05c2a0 0x7ff34c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:07.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.102+0000 7ff36259c700 1 --2- 172.21.15.139:0/9510953 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff34c05c2a0 0x7ff34c05e760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7ff35400b3b0 tx=0x7ff354004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:07.096 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.106+0000 7ff352ffd700 1 -- 172.21.15.139:0/9510953 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff358049270 con 0x7ff3640f6850 2021-11-20T06:06:07.297 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.312+0000 7ff368fe3700 1 -- 172.21.15.139:0/9510953 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff364027a00 con 0x7ff3640f6850 2021-11-20T06:06:07.297 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.313+0000 7ff352ffd700 1 -- 172.21.15.139:0/9510953 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+2947 (secure 0 0 0) 0x7ff35804c890 con 0x7ff3640f6850 2021-11-20T06:06:07.298 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:07.298 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":53},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":55,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:06.353587+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14564":{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":54,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:07.300 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.314+0000 7ff368fe3700 1 -- 172.21.15.139:0/9510953 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff34c05c2a0 msgr2=0x7ff34c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:07.300 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.314+0000 7ff368fe3700 1 --2- 172.21.15.139:0/9510953 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff34c05c2a0 0x7ff34c05e760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7ff35400b3b0 tx=0x7ff354004d10).stop 2021-11-20T06:06:07.301 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.314+0000 7ff368fe3700 1 -- 172.21.15.139:0/9510953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3640f6850 msgr2=0x7ff36410e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:07.301 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.314+0000 7ff368fe3700 1 --2- 172.21.15.139:0/9510953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3640f6850 0x7ff36410e190 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7ff3580026d0 tx=0x7ff358009de0).stop 2021-11-20T06:06:07.301 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.314+0000 7ff368fe3700 1 -- 172.21.15.139:0/9510953 shutdown_connections 2021-11-20T06:06:07.301 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.314+0000 7ff368fe3700 1 --2- 172.21.15.139:0/9510953 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff34c05c2a0 0x7ff34c05e760 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:07.302 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.314+0000 7ff368fe3700 1 --2- 172.21.15.139:0/9510953 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3640f55b0 0x7ff36405ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:07.302 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.314+0000 7ff368fe3700 1 --2- 172.21.15.139:0/9510953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff3640f6850 0x7ff36410e190 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:07.302 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.314+0000 7ff368fe3700 1 -- 172.21.15.139:0/9510953 >> 172.21.15.139:0/9510953 conn(0x7ff3640f0d10 msgr2=0x7ff3640fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:07.302 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.314+0000 7ff368fe3700 1 -- 172.21.15.139:0/9510953 shutdown_connections 2021-11-20T06:06:07.302 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:07.314+0000 7ff368fe3700 1 -- 172.21.15.139:0/9510953 wait complete. 2021-11-20T06:06:07.303 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 55 2021-11-20T06:06:07.312 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:06.353587+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14564}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14564': {'gid': 14564, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 54, 'state': 'up:reconnect', 'state_seq': 6, 'addr': '172.21.15.139:6827/3938771458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 3938771458}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 3938771458}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:07.312 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14564:up:reconnect 2021-11-20T06:06:07.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:07 smithi178 conmon[36600]: cluster 2021-11-20T06:06:06.252483+0000 mgr.smithi139.dplyxn (mgr.14162) 269 : cluster [DBG] 2021-11-20T06:06:07.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:07 smithi178 conmon[36600]: pgmap v239: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2021-11-20T06:06:07.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:07 smithi178 conmon[36600]: cluster 2021-11-20T06:06:06.360310+0000 mon.smithi139 (mon.0) 688 : cluster [DBG] mds.? [v2:172.21.15.139:6826/3938771458,v1:172.21.15.139:6827/3938771458] up:reconnect 2021-11-20T06:06:07.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:07 smithi178 conmon[36600]: cluster 2021-11-20T06:06 2021-11-20T06:06:07.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:07 smithi178 conmon[36600]: :06.360350+0000 mon.smithi139 (mon.0) 689 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:reconnect} 3 up:standby 2021-11-20T06:06:07.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:07 smithi178 conmon[36600]: audit 2021 2021-11-20T06:06:07.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:07 smithi178 conmon[36600]: -11-20T06:06:07.313395+0000 mon.smithi139 (mon.0) 690 : 2021-11-20T06:06:07.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:07 smithi178 conmon[36600]: audit [DBG] from='client.? 172.21.15.139:0/9510953' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:07.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:07 smithi139 conmon[33501]: cluster 2021-11-20T06:06:06.252483+0000 mgr.smithi139.dplyxn (mgr.14162) 269 : cluster [DBG] pgmap v239: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2021-11-20T06:06:07.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:07 smithi139 conmon[33501]: cluster 2021-11-20T06:06:06.360310+0000 mon.smithi139 (mon.0) 688 : cluster [DBG] mds.? [v2:172.21.15.139:6826/3938771458,v1:172.21.15.139:6827/3938771458] up:reconnect 2021-11-20T06:06:07.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:07 smithi139 conmon[33501]: cluster 2021-11-20T06:06:06 2021-11-20T06:06:07.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:07 smithi139 conmon[33501]: .360350+0000 mon.smithi139 (mon.0) 689 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:reconnect} 3 up:standby 2021-11-20T06:06:07.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:07 smithi139 conmon[33501]: audit 2021 2021-11-20T06:06:07.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:07 smithi139 conmon[33501]: -11-20T06:06:07.313395+0000 mon.smithi139 (mon.0) 690 : audit [DBG] 2021-11-20T06:06:07.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:07 smithi139 conmon[33501]: from='client.? 172.21.15.139:0/9510953' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:08.313 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:08.446 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.459+0000 7f1018f85700 1 -- 172.21.15.139:0/3665806797 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10140f5910 msgr2=0x7f10140f5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:08.446 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.459+0000 7f1018f85700 1 --2- 172.21.15.139:0/3665806797 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10140f5910 0x7f10140f5d30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f1008004660 tx=0x7f1008009b30).stop 2021-11-20T06:06:08.447 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.459+0000 7f1018f85700 1 -- 172.21.15.139:0/3665806797 shutdown_connections 2021-11-20T06:06:08.447 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.459+0000 7f1018f85700 1 --2- 172.21.15.139:0/3665806797 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10140f4660 0x7f10140f4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:08.447 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.459+0000 7f1018f85700 1 --2- 172.21.15.139:0/3665806797 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10140f5910 0x7f10140f5d30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:08.447 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.459+0000 7f1018f85700 1 -- 172.21.15.139:0/3665806797 >> 172.21.15.139:0/3665806797 conn(0x7f10140f0360 msgr2=0x7f10140f27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:08.448 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.459+0000 7f1018f85700 1 -- 172.21.15.139:0/3665806797 shutdown_connections 2021-11-20T06:06:08.448 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.459+0000 7f1018f85700 1 -- 172.21.15.139:0/3665806797 wait complete. 2021-11-20T06:06:08.448 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.460+0000 7f1018f85700 1 Processor -- start 2021-11-20T06:06:08.448 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.460+0000 7f1018f85700 1 -- start start 2021-11-20T06:06:08.449 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.460+0000 7f1018f85700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10140f4660 0x7f101410d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:08.450 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.460+0000 7f1018f85700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10140f5910 0x7f101410d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:08.450 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.460+0000 7f1018f85700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101410de80 con 0x7f10140f5910 2021-11-20T06:06:08.450 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.460+0000 7f1018f85700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101410e730 con 0x7f10140f4660 2021-11-20T06:06:08.450 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.460+0000 7f1012ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10140f5910 0x7f101410d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:08.451 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.460+0000 7f1012ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10140f5910 0x7f101410d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58106/0 (socket says 172.21.15.139:58106) 2021-11-20T06:06:08.451 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.460+0000 7f1012ffd700 1 -- 172.21.15.139:0/3474301987 learned_addr learned my addr 172.21.15.139:0/3474301987 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:08.451 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.460+0000 7f10137fe700 1 --2- 172.21.15.139:0/3474301987 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10140f4660 0x7f101410d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:08.451 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.461+0000 7f1012ffd700 1 -- 172.21.15.139:0/3474301987 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10140f4660 msgr2=0x7f101410d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:08.451 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.461+0000 7f1012ffd700 1 --2- 172.21.15.139:0/3474301987 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10140f4660 0x7f101410d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:08.452 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.461+0000 7f1012ffd700 1 -- 172.21.15.139:0/3474301987 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1008005040 con 0x7f10140f5910 2021-11-20T06:06:08.452 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.461+0000 7f1012ffd700 1 --2- 172.21.15.139:0/3474301987 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10140f5910 0x7f101410d840 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f100400b410 tx=0x7f1004006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:08.452 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.461+0000 7f0ffbfff700 1 -- 172.21.15.139:0/3474301987 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10040125f0 con 0x7f10140f5910 2021-11-20T06:06:08.452 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.461+0000 7f1018f85700 1 -- 172.21.15.139:0/3474301987 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101410ea10 con 0x7f10140f5910 2021-11-20T06:06:08.453 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.461+0000 7f0ffbfff700 1 -- 172.21.15.139:0/3474301987 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f100400be30 con 0x7f10140f5910 2021-11-20T06:06:08.453 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.461+0000 7f1018f85700 1 -- 172.21.15.139:0/3474301987 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f101410efd0 con 0x7f10140f5910 2021-11-20T06:06:08.453 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.461+0000 7f0ffbfff700 1 -- 172.21.15.139:0/3474301987 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1004002a20 con 0x7f10140f5910 2021-11-20T06:06:08.454 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.462+0000 7f1018f85700 1 -- 172.21.15.139:0/3474301987 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10140f4010 con 0x7f10140f5910 2021-11-20T06:06:08.455 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.462+0000 7f0ffbfff700 1 -- 172.21.15.139:0/3474301987 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f100400f030 con 0x7f10140f5910 2021-11-20T06:06:08.455 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.463+0000 7f0ffbfff700 1 --2- 172.21.15.139:0/3474301987 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0ffc05c350 0x7f0ffc05e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:08.455 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.463+0000 7f0ffbfff700 1 -- 172.21.15.139:0/3474301987 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f1004076670 con 0x7f10140f5910 2021-11-20T06:06:08.455 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.463+0000 7f10137fe700 1 --2- 172.21.15.139:0/3474301987 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0ffc05c350 0x7f0ffc05e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:08.456 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.463+0000 7f10137fe700 1 --2- 172.21.15.139:0/3474301987 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0ffc05c350 0x7f0ffc05e810 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f1008004630 tx=0x7f1008009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:08.456 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.467+0000 7f0ffbfff700 1 -- 172.21.15.139:0/3474301987 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1004048f70 con 0x7f10140f5910 2021-11-20T06:06:08.468 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:08 smithi139 conmon[33501]: cluster 2021-11-20T06:06:07.368404+0000 mon.smithi139 (mon.0) 691 : cluster [DBG] mds.? [v2:172.21.15.139:6826/3938771458,v1:172.21.15.139:6827/3938771458] up:rejoin 2021-11-20T06:06:08.469 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:08 smithi139 conmon[33501]: cluster 2021-11-20T06:06:07.368464+0000 mon.smithi139 (mon.0) 692 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:06:08.658 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.673+0000 7f1018f85700 1 -- 172.21.15.139:0/3474301987 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1014045ad0 con 0x7f10140f5910 2021-11-20T06:06:08.658 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.673+0000 7f0ffbfff700 1 -- 172.21.15.139:0/3474301987 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2944 (secure 0 0 0) 0x7f100404c590 con 0x7f10140f5910 2021-11-20T06:06:08.659 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:08.659 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":53},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":56,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:07.358398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14564":{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:08.667 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.675+0000 7f1018f85700 1 -- 172.21.15.139:0/3474301987 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0ffc05c350 msgr2=0x7f0ffc05e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:08.667 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.675+0000 7f1018f85700 1 --2- 172.21.15.139:0/3474301987 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0ffc05c350 0x7f0ffc05e810 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f1008004630 tx=0x7f1008009b30).stop 2021-11-20T06:06:08.667 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.675+0000 7f1018f85700 1 -- 172.21.15.139:0/3474301987 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10140f5910 msgr2=0x7f101410d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:08.667 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.675+0000 7f1018f85700 1 --2- 172.21.15.139:0/3474301987 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10140f5910 0x7f101410d840 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f100400b410 tx=0x7f1004006ef0).stop 2021-11-20T06:06:08.668 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.675+0000 7f1018f85700 1 -- 172.21.15.139:0/3474301987 shutdown_connections 2021-11-20T06:06:08.668 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.675+0000 7f1018f85700 1 --2- 172.21.15.139:0/3474301987 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0ffc05c350 0x7f0ffc05e810 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:08.668 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.675+0000 7f1018f85700 1 --2- 172.21.15.139:0/3474301987 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10140f4660 0x7f101410d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:08.668 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.675+0000 7f1018f85700 1 --2- 172.21.15.139:0/3474301987 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f10140f5910 0x7f101410d840 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:08.669 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.675+0000 7f1018f85700 1 -- 172.21.15.139:0/3474301987 >> 172.21.15.139:0/3474301987 conn(0x7f10140f0360 msgr2=0x7f10140f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:08.669 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.675+0000 7f1018f85700 1 -- 172.21.15.139:0/3474301987 shutdown_connections 2021-11-20T06:06:08.669 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:08.675+0000 7f1018f85700 1 -- 172.21.15.139:0/3474301987 wait complete. 2021-11-20T06:06:08.669 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 56 2021-11-20T06:06:08.672 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:07.358398+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14564}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14564': {'gid': 14564, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 54, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.139:6827/3938771458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 3938771458}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 3938771458}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:08.673 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14564:up:rejoin 2021-11-20T06:06:08.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:08 smithi178 conmon[36600]: cluster 2021-11-20T06:06:07.368404+0000 mon.smithi139 (mon.0) 691 : cluster [DBG] 2021-11-20T06:06:08.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:08 smithi178 conmon[36600]: mds.? [v2:172.21.15.139:6826/3938771458,v1:172.21.15.139:6827/3938771458] up:rejoin 2021-11-20T06:06:08.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:08 smithi178 conmon[36600]: cluster 2021-11-20T06:06:07.368464+0000 mon.smithi139 (mon.0) 692 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:06:09.673 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:09.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:09 smithi178 conmon[36600]: cluster 2021-11-20T06:06:08.253147+0000 mgr.smithi139.dplyxn (mgr.14162) 270 : cluster [DBG] pgmap v240: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-20T06:06:09.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:09 smithi178 conmon[36600]: audit 2021-11-20T06:06 2021-11-20T06:06:09.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:09 smithi178 conmon[36600]: :08.674294+0000 mon.smithi139 (mon.0) 693 : audit [DBG] from='client.? 172.21.15.139:0/3474301987' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:09.701 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:09 smithi139 conmon[33501]: cluster 2021-11-20T06:06:08.253147+0000 mgr.smithi139.dplyxn (mgr.14162) 270 : cluster 2021-11-20T06:06:09.701 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:09 smithi139 conmon[33501]: [DBG] pgmap v240: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-20T06:06:09.702 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:09 smithi139 conmon[33501]: audit 2021-11-20T06:06:08.674294+0000 mon.smithi139 (mon.0) 693 : audit [DBG] from='client.? 172.21.15.139:0/3474301987' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:09.802 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.815+0000 7f1748246700 1 -- 172.21.15.139:0/3500371000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17400f6840 msgr2=0x7f17400f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:09.802 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.815+0000 7f1748246700 1 --2- 172.21.15.139:0/3500371000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17400f6840 0x7f17400f6ca0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f1730004660 tx=0x7f1730009b30).stop 2021-11-20T06:06:09.802 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.815+0000 7f1748246700 1 -- 172.21.15.139:0/3500371000 shutdown_connections 2021-11-20T06:06:09.803 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.815+0000 7f1748246700 1 --2- 172.21.15.139:0/3500371000 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17400f6840 0x7f17400f6ca0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:09.803 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.815+0000 7f1748246700 1 --2- 172.21.15.139:0/3500371000 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17400f55a0 0x7f17400f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:09.803 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.815+0000 7f1748246700 1 -- 172.21.15.139:0/3500371000 >> 172.21.15.139:0/3500371000 conn(0x7f17400f0ce0 msgr2=0x7f17400f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:09.803 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.815+0000 7f1748246700 1 -- 172.21.15.139:0/3500371000 shutdown_connections 2021-11-20T06:06:09.804 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.815+0000 7f1748246700 1 -- 172.21.15.139:0/3500371000 wait complete. 2021-11-20T06:06:09.804 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.816+0000 7f1748246700 1 Processor -- start 2021-11-20T06:06:09.804 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.816+0000 7f1748246700 1 -- start start 2021-11-20T06:06:09.804 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.816+0000 7f1748246700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17400f55a0 0x7f1740106e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:09.805 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.816+0000 7f1748246700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17400f6840 0x7f1740107350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:09.805 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.816+0000 7f1748246700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1740107900 con 0x7f17400f6840 2021-11-20T06:06:09.805 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.816+0000 7f1748246700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1740107a40 con 0x7f17400f55a0 2021-11-20T06:06:09.805 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.816+0000 7f17457e1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17400f6840 0x7f1740107350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:09.806 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.816+0000 7f17457e1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17400f6840 0x7f1740107350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58120/0 (socket says 172.21.15.139:58120) 2021-11-20T06:06:09.806 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.816+0000 7f17457e1700 1 -- 172.21.15.139:0/2078578206 learned_addr learned my addr 172.21.15.139:0/2078578206 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:09.806 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.816+0000 7f17457e1700 1 -- 172.21.15.139:0/2078578206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17400f55a0 msgr2=0x7f1740106e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:09.806 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.816+0000 7f17457e1700 1 --2- 172.21.15.139:0/2078578206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17400f55a0 0x7f1740106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:09.806 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.816+0000 7f17457e1700 1 -- 172.21.15.139:0/2078578206 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1730005040 con 0x7f17400f6840 2021-11-20T06:06:09.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.817+0000 7f17457e1700 1 --2- 172.21.15.139:0/2078578206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17400f6840 0x7f1740107350 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f1730000c00 tx=0x7f173000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:09.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.817+0000 7f17367fc700 1 -- 172.21.15.139:0/2078578206 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1730019070 con 0x7f17400f6840 2021-11-20T06:06:09.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.817+0000 7f17367fc700 1 -- 172.21.15.139:0/2078578206 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1730007e70 con 0x7f17400f6840 2021-11-20T06:06:09.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.817+0000 7f1748246700 1 -- 172.21.15.139:0/2078578206 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f174010c480 con 0x7f17400f6840 2021-11-20T06:06:09.808 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.817+0000 7f17367fc700 1 -- 172.21.15.139:0/2078578206 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1730014d00 con 0x7f17400f6840 2021-11-20T06:06:09.808 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.817+0000 7f1748246700 1 -- 172.21.15.139:0/2078578206 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174010ca40 con 0x7f17400f6840 2021-11-20T06:06:09.809 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.818+0000 7f1748246700 1 -- 172.21.15.139:0/2078578206 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17400fdd50 con 0x7f17400f6840 2021-11-20T06:06:09.809 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.818+0000 7f17367fc700 1 -- 172.21.15.139:0/2078578206 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1730025070 con 0x7f17400f6840 2021-11-20T06:06:09.810 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.818+0000 7f17367fc700 1 --2- 172.21.15.139:0/2078578206 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f172c05c2a0 0x7f172c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:09.810 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.818+0000 7f17367fc700 1 -- 172.21.15.139:0/2078578206 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f1730077230 con 0x7f17400f6840 2021-11-20T06:06:09.810 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.818+0000 7f1745fe2700 1 --2- 172.21.15.139:0/2078578206 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f172c05c2a0 0x7f172c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:09.810 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.819+0000 7f1745fe2700 1 --2- 172.21.15.139:0/2078578206 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f172c05c2a0 0x7f172c05e760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f17400f66a0 tx=0x7f173c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:09.811 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:09.822+0000 7f17367fc700 1 -- 172.21.15.139:0/2078578206 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1730049b90 con 0x7f17400f6840 2021-11-20T06:06:09.992 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:10.007+0000 7f1748246700 1 -- 172.21.15.139:0/2078578206 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f174010d460 con 0x7f17400f6840 2021-11-20T06:06:09.993 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:10.008+0000 7f17367fc700 1 -- 172.21.15.139:0/2078578206 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2943 (secure 0 0 0) 0x7f1730023070 con 0x7f17400f6840 2021-11-20T06:06:09.993 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:09.994 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":56,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:07.358398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14564":{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:09.996 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:10.009+0000 7f1748246700 1 -- 172.21.15.139:0/2078578206 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f172c05c2a0 msgr2=0x7f172c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:09.996 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:10.009+0000 7f1748246700 1 --2- 172.21.15.139:0/2078578206 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f172c05c2a0 0x7f172c05e760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f17400f66a0 tx=0x7f173c00b040).stop 2021-11-20T06:06:09.996 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:10.009+0000 7f1748246700 1 -- 172.21.15.139:0/2078578206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17400f6840 msgr2=0x7f1740107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:09.996 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:10.009+0000 7f1748246700 1 --2- 172.21.15.139:0/2078578206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17400f6840 0x7f1740107350 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f1730000c00 tx=0x7f173000b040).stop 2021-11-20T06:06:09.997 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:10.009+0000 7f1748246700 1 -- 172.21.15.139:0/2078578206 shutdown_connections 2021-11-20T06:06:09.997 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:10.009+0000 7f1748246700 1 --2- 172.21.15.139:0/2078578206 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f172c05c2a0 0x7f172c05e760 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:09.997 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:10.009+0000 7f1748246700 1 --2- 172.21.15.139:0/2078578206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17400f55a0 0x7f1740106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:09.997 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:10.009+0000 7f1748246700 1 --2- 172.21.15.139:0/2078578206 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f17400f6840 0x7f1740107350 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:09.998 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:10.009+0000 7f1748246700 1 -- 172.21.15.139:0/2078578206 >> 172.21.15.139:0/2078578206 conn(0x7f17400f0ce0 msgr2=0x7f174005f450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:09.998 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:10.009+0000 7f1748246700 1 -- 172.21.15.139:0/2078578206 shutdown_connections 2021-11-20T06:06:09.999 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:10.009+0000 7f1748246700 1 -- 172.21.15.139:0/2078578206 wait complete. 2021-11-20T06:06:09.999 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 57 2021-11-20T06:06:10.007 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:07.358398+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14564}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14564': {'gid': 14564, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 54, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.139:6827/3938771458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 3938771458}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 3938771458}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:10.008 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14564:up:rejoin 2021-11-20T06:06:10.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:10 smithi178 conmon[36600]: cluster 2021-11-20T06:06:09.377111+0000 mon.smithi139 (mon.0) 694 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2991819277,v1:172.21.15.139:6829/2991819277] up:standby 2021-11-20T06:06:10.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:10 smithi178 conmon[36600]: cluster 2021-11-20T06:06:09 2021-11-20T06:06:10.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:10 smithi178 conmon[36600]: .377176+0000 mon.smithi139 (mon.0) 695 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:06:10.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:10 smithi178 conmon[36600]: audit 2021-11-20T06:06:10.008413+0000 mon.smithi139 (mon.0) 696 : audit [DBG] from='client.? 172.21.15.139:0/2078578206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:10.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:10 smithi139 conmon[33501]: cluster 2021-11-20T06:06:09.377111+0000 mon.smithi139 (mon.0) 694 2021-11-20T06:06:10.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:10 smithi139 conmon[33501]: : cluster [DBG] mds.? [v2:172.21.15.139:6828/2991819277,v1:172.21.15.139:6829/2991819277] up:standby 2021-11-20T06:06:10.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:10 smithi139 conmon[33501]: cluster 2021-11-20T06:06:09.377176+0000 mon.smithi139 (mon.0) 695 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 3 up:standby 2021-11-20T06:06:10.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:10 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:06:10.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:10 smithi139 conmon[33501]: 20T06:06:10.008413+0000 mon.smithi139 (mon.0) 696 : audit [DBG] from='client.? 172.21.15.139:0/2078578206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:11.009 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:11.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.150+0000 7f7b58f15700 1 -- 172.21.15.139:0/4021804091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b540f7960 msgr2=0x7f7b540f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:11.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.150+0000 7f7b58f15700 1 --2- 172.21.15.139:0/4021804091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b540f7960 0x7f7b540f7dc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f7b48004660 tx=0x7f7b48009b30).stop 2021-11-20T06:06:11.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.151+0000 7f7b58f15700 1 -- 172.21.15.139:0/4021804091 shutdown_connections 2021-11-20T06:06:11.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.151+0000 7f7b58f15700 1 --2- 172.21.15.139:0/4021804091 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b540f7960 0x7f7b540f7dc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:11.139 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.151+0000 7f7b58f15700 1 --2- 172.21.15.139:0/4021804091 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b540f66c0 0x7f7b540f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:11.139 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.151+0000 7f7b58f15700 1 -- 172.21.15.139:0/4021804091 >> 172.21.15.139:0/4021804091 conn(0x7f7b540f1de0 msgr2=0x7f7b540f4240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:11.139 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.151+0000 7f7b58f15700 1 -- 172.21.15.139:0/4021804091 shutdown_connections 2021-11-20T06:06:11.139 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.151+0000 7f7b58f15700 1 -- 172.21.15.139:0/4021804091 wait complete. 2021-11-20T06:06:11.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.151+0000 7f7b58f15700 1 Processor -- start 2021-11-20T06:06:11.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.151+0000 7f7b58f15700 1 -- start start 2021-11-20T06:06:11.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.151+0000 7f7b58f15700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b540f66c0 0x7f7b54106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:11.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.151+0000 7f7b58f15700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b540f7960 0x7f7b541073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:11.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.151+0000 7f7b58f15700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b54107950 con 0x7f7b540f66c0 2021-11-20T06:06:11.141 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.151+0000 7f7b58f15700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b54107a90 con 0x7f7b540f7960 2021-11-20T06:06:11.141 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.151+0000 7f7b5259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b540f66c0 0x7f7b54106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:11.141 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.151+0000 7f7b5259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b540f66c0 0x7f7b54106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58130/0 (socket says 172.21.15.139:58130) 2021-11-20T06:06:11.141 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.151+0000 7f7b5259c700 1 -- 172.21.15.139:0/4167224567 learned_addr learned my addr 172.21.15.139:0/4167224567 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:11.142 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.152+0000 7f7b5259c700 1 -- 172.21.15.139:0/4167224567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b540f7960 msgr2=0x7f7b541073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:11.142 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.152+0000 7f7b5259c700 1 --2- 172.21.15.139:0/4167224567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b540f7960 0x7f7b541073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:11.142 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.152+0000 7f7b5259c700 1 -- 172.21.15.139:0/4167224567 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b48005040 con 0x7f7b540f66c0 2021-11-20T06:06:11.142 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.152+0000 7f7b5259c700 1 --2- 172.21.15.139:0/4167224567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b540f66c0 0x7f7b54106eb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f7b4400c530 tx=0x7f7b44007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:11.142 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.152+0000 7f7b42ffd700 1 -- 172.21.15.139:0/4167224567 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b4400f650 con 0x7f7b540f66c0 2021-11-20T06:06:11.143 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.152+0000 7f7b42ffd700 1 -- 172.21.15.139:0/4167224567 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b4400fc60 con 0x7f7b540f66c0 2021-11-20T06:06:11.143 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.152+0000 7f7b58f15700 1 -- 172.21.15.139:0/4167224567 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5410c530 con 0x7f7b540f66c0 2021-11-20T06:06:11.143 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.152+0000 7f7b42ffd700 1 -- 172.21.15.139:0/4167224567 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b44012d20 con 0x7f7b540f66c0 2021-11-20T06:06:11.143 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.152+0000 7f7b58f15700 1 -- 172.21.15.139:0/4167224567 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b5410caf0 con 0x7f7b540f66c0 2021-11-20T06:06:11.144 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.153+0000 7f7b58f15700 1 -- 172.21.15.139:0/4167224567 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b54027a00 con 0x7f7b540f66c0 2021-11-20T06:06:11.144 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.153+0000 7f7b42ffd700 1 -- 172.21.15.139:0/4167224567 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7b44019030 con 0x7f7b540f66c0 2021-11-20T06:06:11.145 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.154+0000 7f7b42ffd700 1 --2- 172.21.15.139:0/4167224567 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7b3c05c2a0 0x7f7b3c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:11.145 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.154+0000 7f7b42ffd700 1 -- 172.21.15.139:0/4167224567 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f7b44075ea0 con 0x7f7b540f66c0 2021-11-20T06:06:11.145 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.154+0000 7f7b51d9b700 1 --2- 172.21.15.139:0/4167224567 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7b3c05c2a0 0x7f7b3c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:11.146 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.154+0000 7f7b51d9b700 1 --2- 172.21.15.139:0/4167224567 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7b3c05c2a0 0x7f7b3c05e760 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f7b48009f50 tx=0x7f7b48009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:11.146 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.158+0000 7f7b42ffd700 1 -- 172.21.15.139:0/4167224567 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b440488b0 con 0x7f7b540f66c0 2021-11-20T06:06:11.330 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.346+0000 7f7b58f15700 1 -- 172.21.15.139:0/4167224567 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b540fee50 con 0x7f7b540f66c0 2021-11-20T06:06:11.331 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.346+0000 7f7b42ffd700 1 -- 172.21.15.139:0/4167224567 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2943 (secure 0 0 0) 0x7f7b44017090 con 0x7f7b540f66c0 2021-11-20T06:06:11.332 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:11.332 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":56,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:07.358398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14564":{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:11.334 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.348+0000 7f7b58f15700 1 -- 172.21.15.139:0/4167224567 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7b3c05c2a0 msgr2=0x7f7b3c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:11.334 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.348+0000 7f7b58f15700 1 --2- 172.21.15.139:0/4167224567 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7b3c05c2a0 0x7f7b3c05e760 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f7b48009f50 tx=0x7f7b48009b30).stop 2021-11-20T06:06:11.334 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.348+0000 7f7b58f15700 1 -- 172.21.15.139:0/4167224567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b540f66c0 msgr2=0x7f7b54106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:11.334 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.348+0000 7f7b58f15700 1 --2- 172.21.15.139:0/4167224567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b540f66c0 0x7f7b54106eb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f7b4400c530 tx=0x7f7b44007e10).stop 2021-11-20T06:06:11.335 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.348+0000 7f7b58f15700 1 -- 172.21.15.139:0/4167224567 shutdown_connections 2021-11-20T06:06:11.335 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.348+0000 7f7b58f15700 1 --2- 172.21.15.139:0/4167224567 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7b3c05c2a0 0x7f7b3c05e760 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:11.335 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.348+0000 7f7b58f15700 1 --2- 172.21.15.139:0/4167224567 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7b540f66c0 0x7f7b54106eb0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:11.335 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.348+0000 7f7b58f15700 1 --2- 172.21.15.139:0/4167224567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b540f7960 0x7f7b541073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:11.336 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.348+0000 7f7b58f15700 1 -- 172.21.15.139:0/4167224567 >> 172.21.15.139:0/4167224567 conn(0x7f7b540f1de0 msgr2=0x7f7b540fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:11.336 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.348+0000 7f7b58f15700 1 -- 172.21.15.139:0/4167224567 shutdown_connections 2021-11-20T06:06:11.336 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:11.348+0000 7f7b58f15700 1 -- 172.21.15.139:0/4167224567 wait complete. 2021-11-20T06:06:11.336 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 57 2021-11-20T06:06:11.345 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:07.358398+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14564}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14564': {'gid': 14564, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 54, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.139:6827/3938771458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 3938771458}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 3938771458}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:11.345 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14564:up:rejoin 2021-11-20T06:06:11.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:11 smithi178 conmon[36600]: cluster 2021-11-20T06:06:10.253510+0000 mgr.smithi139.dplyxn (mgr.14162) 271 : 2021-11-20T06:06:11.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:11 smithi178 conmon[36600]: cluster [DBG] pgmap v241: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:11.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:11 smithi178 conmon[36600]: audit 2021-11-20T06:06:11.347124+0000 mon.smithi139 (mon.0) 697 : audit [DBG] from='client.? 172.21.15.139:0/4167224567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:11.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:11 smithi139 conmon[33501]: cluster 2021-11-20T06:06:10.253510+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:06:11.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:11 smithi139 conmon[33501]: 271 : cluster [DBG] pgmap v241: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:11.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:11 smithi139 conmon[33501]: audit 2021-11-20T06:06:11.347124+0000 mon.smithi139 (mon.0) 697 : audit [DBG] from='client.? 172.21.15.139:0/4167224567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:12.347 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:12.476 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.489+0000 7f585773e700 1 -- 172.21.15.139:0/3165215417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58500f6840 msgr2=0x7f58500f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:12.477 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.489+0000 7f585773e700 1 --2- 172.21.15.139:0/3165215417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58500f6840 0x7f58500f6ca0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f5840004660 tx=0x7f5840009b30).stop 2021-11-20T06:06:12.477 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.489+0000 7f585773e700 1 -- 172.21.15.139:0/3165215417 shutdown_connections 2021-11-20T06:06:12.477 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.489+0000 7f585773e700 1 --2- 172.21.15.139:0/3165215417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58500f6840 0x7f58500f6ca0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:12.478 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.489+0000 7f585773e700 1 --2- 172.21.15.139:0/3165215417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58500f55a0 0x7f58500f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:12.478 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.489+0000 7f585773e700 1 -- 172.21.15.139:0/3165215417 >> 172.21.15.139:0/3165215417 conn(0x7f58500f0ce0 msgr2=0x7f58500f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:12.478 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.489+0000 7f585773e700 1 -- 172.21.15.139:0/3165215417 shutdown_connections 2021-11-20T06:06:12.478 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.489+0000 7f585773e700 1 -- 172.21.15.139:0/3165215417 wait complete. 2021-11-20T06:06:12.479 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.489+0000 7f585773e700 1 Processor -- start 2021-11-20T06:06:12.479 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f585773e700 1 -- start start 2021-11-20T06:06:12.479 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f585773e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58500f55a0 0x7f585005ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:12.479 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f585773e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58500f6840 0x7f585010bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:12.480 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f585773e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585010c180 con 0x7f58500f6840 2021-11-20T06:06:12.480 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f585773e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585010c2c0 con 0x7f58500f55a0 2021-11-20T06:06:12.480 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f5854cd9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58500f6840 0x7f585010bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:12.480 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f5854cd9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58500f6840 0x7f585010bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58144/0 (socket says 172.21.15.139:58144) 2021-11-20T06:06:12.480 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f5854cd9700 1 -- 172.21.15.139:0/3628676556 learned_addr learned my addr 172.21.15.139:0/3628676556 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:12.481 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f58554da700 1 --2- 172.21.15.139:0/3628676556 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58500f55a0 0x7f585005ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:12.481 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f5854cd9700 1 -- 172.21.15.139:0/3628676556 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58500f55a0 msgr2=0x7f585005ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:12.481 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f5854cd9700 1 --2- 172.21.15.139:0/3628676556 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58500f55a0 0x7f585005ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:12.482 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f5854cd9700 1 -- 172.21.15.139:0/3628676556 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5840005040 con 0x7f58500f6840 2021-11-20T06:06:12.483 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f5854cd9700 1 --2- 172.21.15.139:0/3628676556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58500f6840 0x7f585010bc90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f5840009830 tx=0x7f5840009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:12.483 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f5845ffb700 1 -- 172.21.15.139:0/3628676556 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5840018070 con 0x7f58500f6840 2021-11-20T06:06:12.484 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f5845ffb700 1 -- 172.21.15.139:0/3628676556 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5840013410 con 0x7f58500f6840 2021-11-20T06:06:12.484 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f585773e700 1 -- 172.21.15.139:0/3628676556 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f585010c4c0 con 0x7f58500f6840 2021-11-20T06:06:12.484 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.490+0000 7f5845ffb700 1 -- 172.21.15.139:0/3628676556 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f584001c3c0 con 0x7f58500f6840 2021-11-20T06:06:12.485 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.491+0000 7f585773e700 1 -- 172.21.15.139:0/3628676556 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f585010ca20 con 0x7f58500f6840 2021-11-20T06:06:12.485 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.492+0000 7f585773e700 1 -- 172.21.15.139:0/3628676556 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5850027a00 con 0x7f58500f6840 2021-11-20T06:06:12.486 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.492+0000 7f5845ffb700 1 -- 172.21.15.139:0/3628676556 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f584001d420 con 0x7f58500f6840 2021-11-20T06:06:12.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.492+0000 7f5845ffb700 1 --2- 172.21.15.139:0/3628676556 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f583c05c2a0 0x7f583c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:12.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.492+0000 7f5845ffb700 1 -- 172.21.15.139:0/3628676556 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f5840077fa0 con 0x7f58500f6840 2021-11-20T06:06:12.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.492+0000 7f58554da700 1 --2- 172.21.15.139:0/3628676556 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f583c05c2a0 0x7f583c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:12.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.493+0000 7f58554da700 1 --2- 172.21.15.139:0/3628676556 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f583c05c2a0 0x7f583c05e760 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f58500f66a0 tx=0x7f584c009c00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:12.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.496+0000 7f5845ffb700 1 -- 172.21.15.139:0/3628676556 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f584004a960 con 0x7f58500f6840 2021-11-20T06:06:12.666 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.681+0000 7f585773e700 1 -- 172.21.15.139:0/3628676556 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58500fdd30 con 0x7f58500f6840 2021-11-20T06:06:12.667 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.682+0000 7f5845ffb700 1 -- 172.21.15.139:0/3628676556 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2943 (secure 0 0 0) 0x7f584004df80 con 0x7f58500f6840 2021-11-20T06:06:12.667 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:12.667 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":56,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:07.358398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14564":{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:12.669 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.683+0000 7f585773e700 1 -- 172.21.15.139:0/3628676556 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f583c05c2a0 msgr2=0x7f583c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:12.669 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.683+0000 7f585773e700 1 --2- 172.21.15.139:0/3628676556 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f583c05c2a0 0x7f583c05e760 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f58500f66a0 tx=0x7f584c009c00).stop 2021-11-20T06:06:12.669 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.683+0000 7f585773e700 1 -- 172.21.15.139:0/3628676556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58500f6840 msgr2=0x7f585010bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:12.670 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.683+0000 7f585773e700 1 --2- 172.21.15.139:0/3628676556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58500f6840 0x7f585010bc90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f5840009830 tx=0x7f5840009b30).stop 2021-11-20T06:06:12.670 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.683+0000 7f585773e700 1 -- 172.21.15.139:0/3628676556 shutdown_connections 2021-11-20T06:06:12.670 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.683+0000 7f585773e700 1 --2- 172.21.15.139:0/3628676556 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f583c05c2a0 0x7f583c05e760 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:12.670 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.683+0000 7f585773e700 1 --2- 172.21.15.139:0/3628676556 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58500f55a0 0x7f585005ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:12.671 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.683+0000 7f585773e700 1 --2- 172.21.15.139:0/3628676556 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58500f6840 0x7f585010bc90 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:12.671 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.683+0000 7f585773e700 1 -- 172.21.15.139:0/3628676556 >> 172.21.15.139:0/3628676556 conn(0x7f58500f0ce0 msgr2=0x7f58500fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:12.671 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.683+0000 7f585773e700 1 -- 172.21.15.139:0/3628676556 shutdown_connections 2021-11-20T06:06:12.671 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:12.683+0000 7f585773e700 1 -- 172.21.15.139:0/3628676556 wait complete. 2021-11-20T06:06:12.671 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 57 2021-11-20T06:06:12.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:07.358398+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14564}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14564': {'gid': 14564, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 54, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.139:6827/3938771458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 3938771458}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 3938771458}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:12.680 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14564:up:rejoin 2021-11-20T06:06:13.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:13 smithi139 conmon[33501]: cluster 2021-11-20T06:06:12.254076+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:06:13.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:13 smithi139 conmon[33501]: ) 272 : cluster [DBG] pgmap v242: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:13.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:13 smithi139 conmon[33501]: audit 2021-11-20T06:06:12.682467+0000 mon.smithi139 (mon.0) 698 : audit [DBG] from='client.? 172.21.15.139:0/3628676556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:13.681 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:13.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:13 smithi178 conmon[36600]: cluster 2021-11-20T06:06:12.254076+0000 mgr.smithi139.dplyxn (mgr.14162) 272 : cluster [DBG] pgmap v242: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:13.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:13 smithi178 conmon[36600]: 2021-11-20T06:06:13.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:13 smithi178 conmon[36600]: audit 2021-11-20T06:06:12.682467+0000 mon.smithi139 (mon.0) 698 : audit [DBG] from='client.? 172.21.15.139:0/3628676556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:13.810 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.823+0000 7f9882b6c700 1 -- 172.21.15.139:0/3146041609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f987c0f4950 msgr2=0x7f987c0f4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:13.810 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.823+0000 7f9882b6c700 1 --2- 172.21.15.139:0/3146041609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f987c0f4950 0x7f987c0f4d70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f986c004660 tx=0x7f986c009b30).stop 2021-11-20T06:06:13.810 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.823+0000 7f9882b6c700 1 -- 172.21.15.139:0/3146041609 shutdown_connections 2021-11-20T06:06:13.810 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.823+0000 7f9882b6c700 1 --2- 172.21.15.139:0/3146041609 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f987c0f52f0 0x7f987c0f7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:13.811 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.823+0000 7f9882b6c700 1 --2- 172.21.15.139:0/3146041609 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f987c0f4950 0x7f987c0f4d70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:13.811 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.823+0000 7f9882b6c700 1 -- 172.21.15.139:0/3146041609 >> 172.21.15.139:0/3146041609 conn(0x7f987c0f0410 msgr2=0x7f987c0f2890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:13.811 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.823+0000 7f9882b6c700 1 -- 172.21.15.139:0/3146041609 shutdown_connections 2021-11-20T06:06:13.811 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.823+0000 7f9882b6c700 1 -- 172.21.15.139:0/3146041609 wait complete. 2021-11-20T06:06:13.812 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.823+0000 7f9882b6c700 1 Processor -- start 2021-11-20T06:06:13.812 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.824+0000 7f9882b6c700 1 -- start start 2021-11-20T06:06:13.812 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.824+0000 7f9882b6c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f987c0f4950 0x7f987c0f7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:13.812 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.824+0000 7f9882b6c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f987c0f52f0 0x7f987c0f5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:13.813 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.824+0000 7f9882b6c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0f78d0 con 0x7f987c0f52f0 2021-11-20T06:06:13.813 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.824+0000 7f9882b6c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0f5e70 con 0x7f987c0f4950 2021-11-20T06:06:13.813 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.824+0000 7f987bfff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f987c0f52f0 0x7f987c0f5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:13.813 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.824+0000 7f987bfff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f987c0f52f0 0x7f987c0f5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58154/0 (socket says 172.21.15.139:58154) 2021-11-20T06:06:13.813 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.824+0000 7f987bfff700 1 -- 172.21.15.139:0/158471007 learned_addr learned my addr 172.21.15.139:0/158471007 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:13.814 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.824+0000 7f9880908700 1 --2- 172.21.15.139:0/158471007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f987c0f4950 0x7f987c0f7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:13.814 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.824+0000 7f987bfff700 1 -- 172.21.15.139:0/158471007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f987c0f4950 msgr2=0x7f987c0f7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:13.814 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.824+0000 7f987bfff700 1 --2- 172.21.15.139:0/158471007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f987c0f4950 0x7f987c0f7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:13.814 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.824+0000 7f987bfff700 1 -- 172.21.15.139:0/158471007 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c005040 con 0x7f987c0f52f0 2021-11-20T06:06:13.815 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.824+0000 7f987bfff700 1 --2- 172.21.15.139:0/158471007 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f987c0f52f0 0x7f987c0f5980 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f9870002700 tx=0x7f9870009d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:13.815 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.824+0000 7f98797fa700 1 -- 172.21.15.139:0/158471007 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98700079c0 con 0x7f987c0f52f0 2021-11-20T06:06:13.815 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.825+0000 7f9882b6c700 1 -- 172.21.15.139:0/158471007 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987c0f60f0 con 0x7f987c0f52f0 2021-11-20T06:06:13.815 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.825+0000 7f98797fa700 1 -- 172.21.15.139:0/158471007 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f987000f040 con 0x7f987c0f52f0 2021-11-20T06:06:13.815 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.825+0000 7f98797fa700 1 -- 172.21.15.139:0/158471007 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9870014740 con 0x7f987c0f52f0 2021-11-20T06:06:13.816 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.825+0000 7f9882b6c700 1 -- 172.21.15.139:0/158471007 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987c0f66b0 con 0x7f987c0f52f0 2021-11-20T06:06:13.817 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.826+0000 7f9882b6c700 1 -- 172.21.15.139:0/158471007 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987c027a00 con 0x7f987c0f52f0 2021-11-20T06:06:13.817 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.826+0000 7f98797fa700 1 -- 172.21.15.139:0/158471007 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f987001b060 con 0x7f987c0f52f0 2021-11-20T06:06:13.818 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.826+0000 7f98797fa700 1 --2- 172.21.15.139:0/158471007 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f986405c2a0 0x7f986405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:13.818 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.826+0000 7f98797fa700 1 -- 172.21.15.139:0/158471007 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f9870076080 con 0x7f987c0f52f0 2021-11-20T06:06:13.818 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.826+0000 7f9880908700 1 --2- 172.21.15.139:0/158471007 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f986405c2a0 0x7f986405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:13.819 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.827+0000 7f9880908700 1 --2- 172.21.15.139:0/158471007 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f986405c2a0 0x7f986405e760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f986c009f50 tx=0x7f986c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:13.819 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:13.830+0000 7f98797fa700 1 -- 172.21.15.139:0/158471007 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98700489e0 con 0x7f987c0f52f0 2021-11-20T06:06:14.001 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:14.016+0000 7f9882b6c700 1 -- 172.21.15.139:0/158471007 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f987c0fee60 con 0x7f987c0f52f0 2021-11-20T06:06:14.002 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:14.017+0000 7f98797fa700 1 -- 172.21.15.139:0/158471007 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2943 (secure 0 0 0) 0x7f987004c000 con 0x7f987c0f52f0 2021-11-20T06:06:14.002 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:14.003 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":56,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:07.358398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14564":{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:14.004 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:14.019+0000 7f9882b6c700 1 -- 172.21.15.139:0/158471007 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f986405c2a0 msgr2=0x7f986405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:14.004 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:14.019+0000 7f9882b6c700 1 --2- 172.21.15.139:0/158471007 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f986405c2a0 0x7f986405e760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f986c009f50 tx=0x7f986c009b30).stop 2021-11-20T06:06:14.005 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:14.019+0000 7f9882b6c700 1 -- 172.21.15.139:0/158471007 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f987c0f52f0 msgr2=0x7f987c0f5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:14.005 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:14.019+0000 7f9882b6c700 1 --2- 172.21.15.139:0/158471007 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f987c0f52f0 0x7f987c0f5980 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f9870002700 tx=0x7f9870009d60).stop 2021-11-20T06:06:14.005 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:14.019+0000 7f9882b6c700 1 -- 172.21.15.139:0/158471007 shutdown_connections 2021-11-20T06:06:14.005 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:14.019+0000 7f9882b6c700 1 --2- 172.21.15.139:0/158471007 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f986405c2a0 0x7f986405e760 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:14.005 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:14.019+0000 7f9882b6c700 1 --2- 172.21.15.139:0/158471007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f987c0f4950 0x7f987c0f7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:14.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:14.019+0000 7f9882b6c700 1 --2- 172.21.15.139:0/158471007 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f987c0f52f0 0x7f987c0f5980 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:14.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:14.019+0000 7f9882b6c700 1 -- 172.21.15.139:0/158471007 >> 172.21.15.139:0/158471007 conn(0x7f987c0f0410 msgr2=0x7f987c0fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:14.007 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:14.019+0000 7f9882b6c700 1 -- 172.21.15.139:0/158471007 shutdown_connections 2021-11-20T06:06:14.008 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:14.019+0000 7f9882b6c700 1 -- 172.21.15.139:0/158471007 wait complete. 2021-11-20T06:06:14.009 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 57 2021-11-20T06:06:14.016 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:07.358398+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14564}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14564': {'gid': 14564, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 54, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.139:6827/3938771458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 3938771458}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 3938771458}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:14.016 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14564:up:rejoin 2021-11-20T06:06:14.277 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:14 smithi139 conmon[33501]: audit 2021-11-20T06:06:14.017843+0000 2021-11-20T06:06:14.277 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:14 smithi139 conmon[33501]: mon.smithi139 (mon.0) 699 : audit [DBG] from='client.? 172.21.15.139:0/158471007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:14.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:14 smithi178 conmon[36600]: audit 2021-11-20T06:06:14.017843+0000 mon.smithi139 (mon.0) 699 : audit [DBG] from='client.? 172.21.15.139:0/158471007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:15.017 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:15.145 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.157+0000 7f3704a60700 1 -- 172.21.15.139:0/2744593223 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37000f66a0 msgr2=0x7f37000f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:15.145 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.157+0000 7f3704a60700 1 --2- 172.21.15.139:0/2744593223 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37000f66a0 0x7f37000f6ac0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f36e8004660 tx=0x7f36e8009b30).stop 2021-11-20T06:06:15.145 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.157+0000 7f3704a60700 1 -- 172.21.15.139:0/2744593223 shutdown_connections 2021-11-20T06:06:15.145 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.157+0000 7f3704a60700 1 --2- 172.21.15.139:0/2744593223 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37000f7940 0x7f37000f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:15.146 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.157+0000 7f3704a60700 1 --2- 172.21.15.139:0/2744593223 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37000f66a0 0x7f37000f6ac0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:15.146 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.157+0000 7f3704a60700 1 -- 172.21.15.139:0/2744593223 >> 172.21.15.139:0/2744593223 conn(0x7f37000f1da0 msgr2=0x7f37000f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:15.146 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.157+0000 7f3704a60700 1 -- 172.21.15.139:0/2744593223 shutdown_connections 2021-11-20T06:06:15.146 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.157+0000 7f3704a60700 1 -- 172.21.15.139:0/2744593223 wait complete. 2021-11-20T06:06:15.147 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.157+0000 7f3704a60700 1 Processor -- start 2021-11-20T06:06:15.147 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.157+0000 7f3704a60700 1 -- start start 2021-11-20T06:06:15.147 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f3704a60700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37000f66a0 0x7f3700106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:15.147 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f3704a60700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37000f7940 0x7f3700107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:15.147 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f3704a60700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3700107940 con 0x7f37000f66a0 2021-11-20T06:06:15.148 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f3704a60700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3700107a80 con 0x7f37000f7940 2021-11-20T06:06:15.148 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f36fe59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37000f66a0 0x7f3700106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:15.148 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f36fe59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37000f66a0 0x7f3700106ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58162/0 (socket says 172.21.15.139:58162) 2021-11-20T06:06:15.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f36fe59c700 1 -- 172.21.15.139:0/3252100077 learned_addr learned my addr 172.21.15.139:0/3252100077 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:15.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f36fdd9b700 1 --2- 172.21.15.139:0/3252100077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37000f7940 0x7f3700107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:15.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f36fe59c700 1 -- 172.21.15.139:0/3252100077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37000f7940 msgr2=0x7f3700107390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:15.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f36fe59c700 1 --2- 172.21.15.139:0/3252100077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37000f7940 0x7f3700107390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:15.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f36fe59c700 1 -- 172.21.15.139:0/3252100077 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e8005040 con 0x7f37000f66a0 2021-11-20T06:06:15.150 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f36fdd9b700 1 --2- 172.21.15.139:0/3252100077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37000f7940 0x7f3700107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:06:15.150 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f36fe59c700 1 --2- 172.21.15.139:0/3252100077 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37000f66a0 0x7f3700106ea0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f36e8004e30 tx=0x7f36e8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:15.150 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f36f6ffd700 1 -- 172.21.15.139:0/3252100077 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36e8018070 con 0x7f37000f66a0 2021-11-20T06:06:15.150 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f36f6ffd700 1 -- 172.21.15.139:0/3252100077 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36e8013410 con 0x7f37000f66a0 2021-11-20T06:06:15.151 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f3704a60700 1 -- 172.21.15.139:0/3252100077 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f370010c4c0 con 0x7f37000f66a0 2021-11-20T06:06:15.151 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f36f6ffd700 1 -- 172.21.15.139:0/3252100077 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36e801c3c0 con 0x7f37000f66a0 2021-11-20T06:06:15.151 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.158+0000 7f3704a60700 1 -- 172.21.15.139:0/3252100077 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f370010ca80 con 0x7f37000f66a0 2021-11-20T06:06:15.151 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.159+0000 7f3704a60700 1 -- 172.21.15.139:0/3252100077 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3700027a00 con 0x7f37000f66a0 2021-11-20T06:06:15.151 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.160+0000 7f36f6ffd700 1 -- 172.21.15.139:0/3252100077 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f36e801d420 con 0x7f37000f66a0 2021-11-20T06:06:15.152 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.160+0000 7f36f6ffd700 1 --2- 172.21.15.139:0/3252100077 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f36ec05c2a0 0x7f36ec05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:15.152 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.160+0000 7f36f6ffd700 1 -- 172.21.15.139:0/3252100077 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f36e8077e60 con 0x7f37000f66a0 2021-11-20T06:06:15.152 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.160+0000 7f36fdd9b700 1 --2- 172.21.15.139:0/3252100077 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f36ec05c2a0 0x7f36ec05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:15.153 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.160+0000 7f36fdd9b700 1 --2- 172.21.15.139:0/3252100077 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f36ec05c2a0 0x7f36ec05e760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f36f0009740 tx=0x7f36f0009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:15.153 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.164+0000 7f36f6ffd700 1 -- 172.21.15.139:0/3252100077 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f36e804a820 con 0x7f37000f66a0 2021-11-20T06:06:15.336 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.351+0000 7f3704a60700 1 -- 172.21.15.139:0/3252100077 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37000fee30 con 0x7f37000f66a0 2021-11-20T06:06:15.337 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.352+0000 7f36f6ffd700 1 -- 172.21.15.139:0/3252100077 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2943 (secure 0 0 0) 0x7f36e804de40 con 0x7f37000f66a0 2021-11-20T06:06:15.337 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:15.338 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":56,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:07.358398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14564":{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:15.339 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.353+0000 7f3704a60700 1 -- 172.21.15.139:0/3252100077 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f36ec05c2a0 msgr2=0x7f36ec05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:15.339 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.353+0000 7f3704a60700 1 --2- 172.21.15.139:0/3252100077 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f36ec05c2a0 0x7f36ec05e760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f36f0009740 tx=0x7f36f0009be0).stop 2021-11-20T06:06:15.340 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.354+0000 7f3704a60700 1 -- 172.21.15.139:0/3252100077 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37000f66a0 msgr2=0x7f3700106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:15.340 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.354+0000 7f3704a60700 1 --2- 172.21.15.139:0/3252100077 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37000f66a0 0x7f3700106ea0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f36e8004e30 tx=0x7f36e8009b30).stop 2021-11-20T06:06:15.340 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.354+0000 7f3704a60700 1 -- 172.21.15.139:0/3252100077 shutdown_connections 2021-11-20T06:06:15.340 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.354+0000 7f3704a60700 1 --2- 172.21.15.139:0/3252100077 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f36ec05c2a0 0x7f36ec05e760 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:15.341 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.354+0000 7f3704a60700 1 --2- 172.21.15.139:0/3252100077 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37000f66a0 0x7f3700106ea0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:15.341 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.354+0000 7f3704a60700 1 --2- 172.21.15.139:0/3252100077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37000f7940 0x7f3700107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:15.341 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.354+0000 7f3704a60700 1 -- 172.21.15.139:0/3252100077 >> 172.21.15.139:0/3252100077 conn(0x7f37000f1da0 msgr2=0x7f37000fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:15.341 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.354+0000 7f3704a60700 1 -- 172.21.15.139:0/3252100077 shutdown_connections 2021-11-20T06:06:15.341 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:15.354+0000 7f3704a60700 1 -- 172.21.15.139:0/3252100077 wait complete. 2021-11-20T06:06:15.342 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 57 2021-11-20T06:06:15.349 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:15 smithi139 conmon[33501]: cluster 2021-11-20T06:06:14.254458+0000 mgr.smithi139.dplyxn (mgr.14162) 273 2021-11-20T06:06:15.349 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:15 smithi139 conmon[33501]: : cluster [DBG] pgmap v243: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:15.350 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:07.358398+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14564}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14564': {'gid': 14564, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 54, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.139:6827/3938771458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 3938771458}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 3938771458}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:15.351 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14564:up:rejoin 2021-11-20T06:06:15.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:15 smithi178 conmon[36600]: cluster 2021-11-20T06:06:14.254458+0000 mgr.smithi139.dplyxn (mgr.14162) 273 : cluster 2021-11-20T06:06:15.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:15 smithi178 conmon[36600]: [DBG] pgmap v243: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:16.352 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:16.485 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.496+0000 7fd21e71f700 1 -- 172.21.15.139:0/2323606661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2180f55a0 msgr2=0x7fd2180f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:16.485 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.496+0000 7fd21e71f700 1 --2- 172.21.15.139:0/2323606661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2180f55a0 0x7fd2180f59c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fd208004660 tx=0x7fd208009b30).stop 2021-11-20T06:06:16.485 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.497+0000 7fd21e71f700 1 -- 172.21.15.139:0/2323606661 shutdown_connections 2021-11-20T06:06:16.485 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.497+0000 7fd21e71f700 1 --2- 172.21.15.139:0/2323606661 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2180f6840 0x7fd2180f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:16.486 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.497+0000 7fd21e71f700 1 --2- 172.21.15.139:0/2323606661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2180f55a0 0x7fd2180f59c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:16.486 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.497+0000 7fd21e71f700 1 -- 172.21.15.139:0/2323606661 >> 172.21.15.139:0/2323606661 conn(0x7fd2180f0ce0 msgr2=0x7fd2180f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:16.486 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.497+0000 7fd21e71f700 1 -- 172.21.15.139:0/2323606661 shutdown_connections 2021-11-20T06:06:16.486 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.497+0000 7fd21e71f700 1 -- 172.21.15.139:0/2323606661 wait complete. 2021-11-20T06:06:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.497+0000 7fd21e71f700 1 Processor -- start 2021-11-20T06:06:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.497+0000 7fd21e71f700 1 -- start start 2021-11-20T06:06:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.497+0000 7fd21e71f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2180f55a0 0x7fd218105dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:16.487 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.497+0000 7fd21e71f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2180f6840 0x7fd2181062c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.497+0000 7fd21e71f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd218106870 con 0x7fd2180f6840 2021-11-20T06:06:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.497+0000 7fd21e71f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2181069b0 con 0x7fd2180f55a0 2021-11-20T06:06:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.498+0000 7fd2177fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2180f6840 0x7fd2181062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.498+0000 7fd2177fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2180f6840 0x7fd2181062c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58176/0 (socket says 172.21.15.139:58176) 2021-11-20T06:06:16.488 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.498+0000 7fd2177fe700 1 -- 172.21.15.139:0/3017967263 learned_addr learned my addr 172.21.15.139:0/3017967263 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:16.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.498+0000 7fd217fff700 1 --2- 172.21.15.139:0/3017967263 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2180f55a0 0x7fd218105dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:16.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.498+0000 7fd2177fe700 1 -- 172.21.15.139:0/3017967263 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2180f55a0 msgr2=0x7fd218105dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:16.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.498+0000 7fd2177fe700 1 --2- 172.21.15.139:0/3017967263 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2180f55a0 0x7fd218105dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:16.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.498+0000 7fd2177fe700 1 -- 172.21.15.139:0/3017967263 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd208005040 con 0x7fd2180f6840 2021-11-20T06:06:16.490 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.498+0000 7fd2177fe700 1 --2- 172.21.15.139:0/3017967263 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2180f6840 0x7fd2181062c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fd20c00c530 tx=0x7fd20c007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:16.490 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.498+0000 7fd214ff9700 1 -- 172.21.15.139:0/3017967263 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd20c00f650 con 0x7fd2180f6840 2021-11-20T06:06:16.490 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.498+0000 7fd21e71f700 1 -- 172.21.15.139:0/3017967263 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd21810b450 con 0x7fd2180f6840 2021-11-20T06:06:16.490 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.498+0000 7fd214ff9700 1 -- 172.21.15.139:0/3017967263 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd20c00fc60 con 0x7fd2180f6840 2021-11-20T06:06:16.491 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.498+0000 7fd214ff9700 1 -- 172.21.15.139:0/3017967263 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd20c012d30 con 0x7fd2180f6840 2021-11-20T06:06:16.491 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.498+0000 7fd21e71f700 1 -- 172.21.15.139:0/3017967263 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd21810ba10 con 0x7fd2180f6840 2021-11-20T06:06:16.491 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.499+0000 7fd21e71f700 1 -- 172.21.15.139:0/3017967263 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd218027a00 con 0x7fd2180f6840 2021-11-20T06:06:16.491 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.499+0000 7fd214ff9700 1 -- 172.21.15.139:0/3017967263 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd20c010070 con 0x7fd2180f6840 2021-11-20T06:06:16.491 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.500+0000 7fd214ff9700 1 --2- 172.21.15.139:0/3017967263 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd20005c2a0 0x7fd20005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:16.492 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.500+0000 7fd214ff9700 1 -- 172.21.15.139:0/3017967263 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fd20c075f50 con 0x7fd2180f6840 2021-11-20T06:06:16.492 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.500+0000 7fd217fff700 1 --2- 172.21.15.139:0/3017967263 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd20005c2a0 0x7fd20005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:16.492 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.500+0000 7fd217fff700 1 --2- 172.21.15.139:0/3017967263 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd20005c2a0 0x7fd20005e760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fd208009d60 tx=0x7fd208009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:16.493 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.504+0000 7fd214ff9700 1 -- 172.21.15.139:0/3017967263 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd20c0452b0 con 0x7fd2180f6840 2021-11-20T06:06:16.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:16 smithi139 conmon[33501]: audit 2021-11-20T06:06:15.352804+0000 mon.smithi139 (mon.0) 700 : audit [DBG] from='client.? 172.21.15.139:0/3252100077' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:16.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:16 smithi139 conmon[33501]: 2021-11-20T06:06:16.672 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.687+0000 7fd21e71f700 1 -- 172.21.15.139:0/3017967263 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2180fdd30 con 0x7fd2180f6840 2021-11-20T06:06:16.672 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.687+0000 7fd214ff9700 1 -- 172.21.15.139:0/3017967263 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2943 (secure 0 0 0) 0x7fd20c048910 con 0x7fd2180f6840 2021-11-20T06:06:16.672 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:16.673 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":56,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:07.358398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14564":{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:16.674 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.689+0000 7fd21e71f700 1 -- 172.21.15.139:0/3017967263 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd20005c2a0 msgr2=0x7fd20005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:16.675 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.689+0000 7fd21e71f700 1 --2- 172.21.15.139:0/3017967263 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd20005c2a0 0x7fd20005e760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fd208009d60 tx=0x7fd208009b30).stop 2021-11-20T06:06:16.675 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.689+0000 7fd21e71f700 1 -- 172.21.15.139:0/3017967263 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2180f6840 msgr2=0x7fd2181062c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:16.675 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.689+0000 7fd21e71f700 1 --2- 172.21.15.139:0/3017967263 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2180f6840 0x7fd2181062c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fd20c00c530 tx=0x7fd20c007e20).stop 2021-11-20T06:06:16.675 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.689+0000 7fd21e71f700 1 -- 172.21.15.139:0/3017967263 shutdown_connections 2021-11-20T06:06:16.676 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.689+0000 7fd21e71f700 1 --2- 172.21.15.139:0/3017967263 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd20005c2a0 0x7fd20005e760 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:16.676 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.689+0000 7fd21e71f700 1 --2- 172.21.15.139:0/3017967263 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2180f55a0 0x7fd218105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:16.676 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.689+0000 7fd21e71f700 1 --2- 172.21.15.139:0/3017967263 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2180f6840 0x7fd2181062c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:16.676 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.689+0000 7fd21e71f700 1 -- 172.21.15.139:0/3017967263 >> 172.21.15.139:0/3017967263 conn(0x7fd2180f0ce0 msgr2=0x7fd2180fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:16.676 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.689+0000 7fd21e71f700 1 -- 172.21.15.139:0/3017967263 shutdown_connections 2021-11-20T06:06:16.677 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:16.689+0000 7fd21e71f700 1 -- 172.21.15.139:0/3017967263 wait complete. 2021-11-20T06:06:16.677 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 57 2021-11-20T06:06:16.687 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:07.358398+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14564}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14564': {'gid': 14564, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 54, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.139:6827/3938771458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 3938771458}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 3938771458}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:16.688 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14564:up:rejoin 2021-11-20T06:06:16.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:16 smithi178 conmon[36600]: audit 2021-11-20T06:06:15.352804+0000 mon.smithi139 (mon.0) 700 : 2021-11-20T06:06:16.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:16 smithi178 conmon[36600]: audit [DBG] from='client.? 172.21.15.139:0/3252100077' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:17.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:17 smithi139 conmon[33501]: cluster 2021-11-20T06:06:16.254891+0000 mgr.smithi139.dplyxn (mgr.14162) 274 : cluster [DBG] 2021-11-20T06:06:17.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:17 smithi139 conmon[33501]: pgmap v244: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:06:17.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:17 smithi139 conmon[33501]: audit 2021-11-20T06:06:16.687978+0000 mon.smithi139 (mon.0) 701 : audit [DBG] from='client.? 172.21.15.139:0/3017967263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:17.689 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:17.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:17 smithi178 conmon[36600]: cluster 2021-11-20T06:06:16.254891+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:06:17.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:17 smithi178 conmon[36600]: ) 274 : cluster [DBG] pgmap v244: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:06:17.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:17 smithi178 conmon[36600]: audit 2021-11-20T06:06:16.687978+0000 mon.smithi139 (mon.0) 701 : audit [DBG] from='client.? 172.21.15.139:0/3017967263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:17.835 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.843+0000 7f3c195f4700 1 -- 172.21.15.139:0/33637015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c140f58f0 msgr2=0x7f3c140f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:17.835 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.843+0000 7f3c195f4700 1 --2- 172.21.15.139:0/33637015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c140f58f0 0x7f3c140f5d70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f3bfc004660 tx=0x7f3bfc009b30).stop 2021-11-20T06:06:17.836 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.843+0000 7f3c195f4700 1 -- 172.21.15.139:0/33637015 shutdown_connections 2021-11-20T06:06:17.836 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.843+0000 7f3c195f4700 1 --2- 172.21.15.139:0/33637015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c140f58f0 0x7f3c140f5d70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:17.837 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.843+0000 7f3c195f4700 1 --2- 172.21.15.139:0/33637015 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c140f6ba0 0x7f3c140f6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:17.837 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.843+0000 7f3c195f4700 1 -- 172.21.15.139:0/33637015 >> 172.21.15.139:0/33637015 conn(0x7f3c140f0440 msgr2=0x7f3c140f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:17.838 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.843+0000 7f3c195f4700 1 -- 172.21.15.139:0/33637015 shutdown_connections 2021-11-20T06:06:17.838 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.843+0000 7f3c195f4700 1 -- 172.21.15.139:0/33637015 wait complete. 2021-11-20T06:06:17.839 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.844+0000 7f3c195f4700 1 Processor -- start 2021-11-20T06:06:17.839 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.844+0000 7f3c195f4700 1 -- start start 2021-11-20T06:06:17.839 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.844+0000 7f3c195f4700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c140f58f0 0x7f3c14105e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:17.840 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.844+0000 7f3c195f4700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c140f6ba0 0x7f3c14106300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:17.840 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.844+0000 7f3c195f4700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c141068b0 con 0x7f3c140f6ba0 2021-11-20T06:06:17.841 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.844+0000 7f3c195f4700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c141069f0 con 0x7f3c140f58f0 2021-11-20T06:06:17.841 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.844+0000 7f3c127fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c140f6ba0 0x7f3c14106300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:17.842 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.844+0000 7f3c127fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c140f6ba0 0x7f3c14106300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58186/0 (socket says 172.21.15.139:58186) 2021-11-20T06:06:17.842 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.844+0000 7f3c127fc700 1 -- 172.21.15.139:0/4248323224 learned_addr learned my addr 172.21.15.139:0/4248323224 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:17.842 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.844+0000 7f3c12ffd700 1 --2- 172.21.15.139:0/4248323224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c140f58f0 0x7f3c14105e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:17.843 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.845+0000 7f3c127fc700 1 -- 172.21.15.139:0/4248323224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c140f58f0 msgr2=0x7f3c14105e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:17.843 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.845+0000 7f3c127fc700 1 --2- 172.21.15.139:0/4248323224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c140f58f0 0x7f3c14105e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:17.844 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.845+0000 7f3c127fc700 1 -- 172.21.15.139:0/4248323224 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc005040 con 0x7f3c140f6ba0 2021-11-20T06:06:17.844 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.845+0000 7f3c127fc700 1 --2- 172.21.15.139:0/4248323224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c140f6ba0 0x7f3c14106300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f3bfc004e30 tx=0x7f3bfc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:17.845 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.845+0000 7f3c0b7fe700 1 -- 172.21.15.139:0/4248323224 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bfc018070 con 0x7f3c140f6ba0 2021-11-20T06:06:17.845 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.845+0000 7f3c195f4700 1 -- 172.21.15.139:0/4248323224 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c140f4840 con 0x7f3c140f6ba0 2021-11-20T06:06:17.846 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.845+0000 7f3c0b7fe700 1 -- 172.21.15.139:0/4248323224 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3bfc013410 con 0x7f3c140f6ba0 2021-11-20T06:06:17.846 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.845+0000 7f3c0b7fe700 1 -- 172.21.15.139:0/4248323224 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bfc01de20 con 0x7f3c140f6ba0 2021-11-20T06:06:17.846 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.845+0000 7f3c195f4700 1 -- 172.21.15.139:0/4248323224 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c140f4e00 con 0x7f3c140f6ba0 2021-11-20T06:06:17.846 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.846+0000 7f3c0b7fe700 1 -- 172.21.15.139:0/4248323224 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3bfc01d420 con 0x7f3c140f6ba0 2021-11-20T06:06:17.847 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.846+0000 7f3c195f4700 1 -- 172.21.15.139:0/4248323224 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c14027a00 con 0x7f3c140f6ba0 2021-11-20T06:06:17.847 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.847+0000 7f3c0b7fe700 1 --2- 172.21.15.139:0/4248323224 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3c0005c2a0 0x7f3c0005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:17.847 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.847+0000 7f3c0b7fe700 1 -- 172.21.15.139:0/4248323224 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f3bfc077d70 con 0x7f3c140f6ba0 2021-11-20T06:06:17.848 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.847+0000 7f3c12ffd700 1 --2- 172.21.15.139:0/4248323224 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3c0005c2a0 0x7f3c0005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:17.848 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.847+0000 7f3c12ffd700 1 --2- 172.21.15.139:0/4248323224 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3c0005c2a0 0x7f3c0005e760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f3c1405e9d0 tx=0x7f3c0c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:17.849 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:17.851+0000 7f3c0b7fe700 1 -- 172.21.15.139:0/4248323224 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3bfc04a7b0 con 0x7f3c140f6ba0 2021-11-20T06:06:18.033 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:18.049+0000 7f3c195f4700 1 -- 172.21.15.139:0/4248323224 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c140fdd70 con 0x7f3c140f6ba0 2021-11-20T06:06:18.034 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:18.049+0000 7f3c0b7fe700 1 -- 172.21.15.139:0/4248323224 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2943 (secure 0 0 0) 0x7f3bfc04ddd0 con 0x7f3c140f6ba0 2021-11-20T06:06:18.035 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:18.035 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":56,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:07.358398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14564":{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:18.037 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:18.052+0000 7f3c195f4700 1 -- 172.21.15.139:0/4248323224 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3c0005c2a0 msgr2=0x7f3c0005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:18.037 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:18.052+0000 7f3c195f4700 1 --2- 172.21.15.139:0/4248323224 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3c0005c2a0 0x7f3c0005e760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f3c1405e9d0 tx=0x7f3c0c009040).stop 2021-11-20T06:06:18.037 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:18.052+0000 7f3c195f4700 1 -- 172.21.15.139:0/4248323224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c140f6ba0 msgr2=0x7f3c14106300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:18.038 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:18.052+0000 7f3c195f4700 1 --2- 172.21.15.139:0/4248323224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c140f6ba0 0x7f3c14106300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f3bfc004e30 tx=0x7f3bfc009b30).stop 2021-11-20T06:06:18.038 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:18.052+0000 7f3c195f4700 1 -- 172.21.15.139:0/4248323224 shutdown_connections 2021-11-20T06:06:18.038 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:18.052+0000 7f3c195f4700 1 --2- 172.21.15.139:0/4248323224 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3c0005c2a0 0x7f3c0005e760 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:18.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:18.052+0000 7f3c195f4700 1 --2- 172.21.15.139:0/4248323224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c140f58f0 0x7f3c14105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:18.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:18.052+0000 7f3c195f4700 1 --2- 172.21.15.139:0/4248323224 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3c140f6ba0 0x7f3c14106300 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:18.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:18.052+0000 7f3c195f4700 1 -- 172.21.15.139:0/4248323224 >> 172.21.15.139:0/4248323224 conn(0x7f3c140f0440 msgr2=0x7f3c140fa350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:18.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:18.052+0000 7f3c195f4700 1 -- 172.21.15.139:0/4248323224 shutdown_connections 2021-11-20T06:06:18.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:18.052+0000 7f3c195f4700 1 -- 172.21.15.139:0/4248323224 wait complete. 2021-11-20T06:06:18.040 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 57 2021-11-20T06:06:18.051 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:07.358398+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14564}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14564': {'gid': 14564, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 54, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.139:6827/3938771458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 3938771458}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 3938771458}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:18.052 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14564:up:rejoin 2021-11-20T06:06:18.225 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:18 smithi139 conmon[33501]: audit 2021-11-20T06:06:18.050198+0000 mon.smithi139 (mon. 2021-11-20T06:06:18.226 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:18 smithi139 conmon[33501]: 0) 702 : audit [DBG] from='client.? 172.21.15.139:0/4248323224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:18.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:18 smithi178 conmon[36600]: audit 2021-11-20T06:06:18.050198+0000 mon.smithi139 (mon.0) 702 : audit 2021-11-20T06:06:18.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:18 smithi178 conmon[36600]: [DBG] from='client.? 172.21.15.139:0/4248323224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:19.053 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:19.184 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.196+0000 7f235b98c700 1 -- 172.21.15.139:0/3277846887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23540f55a0 msgr2=0x7f23540f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:19.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.196+0000 7f235b98c700 1 --2- 172.21.15.139:0/3277846887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23540f55a0 0x7f23540f59c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f23440099e0 tx=0x7f2344004d10).stop 2021-11-20T06:06:19.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.196+0000 7f235b98c700 1 -- 172.21.15.139:0/3277846887 shutdown_connections 2021-11-20T06:06:19.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.196+0000 7f235b98c700 1 --2- 172.21.15.139:0/3277846887 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23540f6840 0x7f23540f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:19.186 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.196+0000 7f235b98c700 1 --2- 172.21.15.139:0/3277846887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23540f55a0 0x7f23540f59c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:19.186 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.196+0000 7f235b98c700 1 -- 172.21.15.139:0/3277846887 >> 172.21.15.139:0/3277846887 conn(0x7f23540f0ce0 msgr2=0x7f23540f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:19.186 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.196+0000 7f235b98c700 1 -- 172.21.15.139:0/3277846887 shutdown_connections 2021-11-20T06:06:19.186 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.196+0000 7f235b98c700 1 -- 172.21.15.139:0/3277846887 wait complete. 2021-11-20T06:06:19.187 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.197+0000 7f235b98c700 1 Processor -- start 2021-11-20T06:06:19.187 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.197+0000 7f235b98c700 1 -- start start 2021-11-20T06:06:19.187 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.197+0000 7f235b98c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23540f55a0 0x7f2354105eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:19.187 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.197+0000 7f235b98c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23540f6840 0x7f23541063a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:19.187 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.197+0000 7f235b98c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2354106950 con 0x7f23540f6840 2021-11-20T06:06:19.188 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.197+0000 7f235b98c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2354106a90 con 0x7f23540f55a0 2021-11-20T06:06:19.188 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.197+0000 7f2358f27700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23540f6840 0x7f23541063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:19.188 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.197+0000 7f2358f27700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23540f6840 0x7f23541063a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58204/0 (socket says 172.21.15.139:58204) 2021-11-20T06:06:19.188 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.197+0000 7f2358f27700 1 -- 172.21.15.139:0/1970741784 learned_addr learned my addr 172.21.15.139:0/1970741784 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:19.189 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.197+0000 7f2359728700 1 --2- 172.21.15.139:0/1970741784 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23540f55a0 0x7f2354105eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:19.189 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.197+0000 7f2358f27700 1 -- 172.21.15.139:0/1970741784 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23540f55a0 msgr2=0x7f2354105eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:19.189 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.197+0000 7f2358f27700 1 --2- 172.21.15.139:0/1970741784 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23540f55a0 0x7f2354105eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:19.189 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.197+0000 7f2358f27700 1 -- 172.21.15.139:0/1970741784 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2344009640 con 0x7f23540f6840 2021-11-20T06:06:19.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.198+0000 7f2358f27700 1 --2- 172.21.15.139:0/1970741784 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23540f6840 0x7f23541063a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f23480026d0 tx=0x7f2348009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:19.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.198+0000 7f2351ffb700 1 -- 172.21.15.139:0/1970741784 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2348009320 con 0x7f23540f6840 2021-11-20T06:06:19.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.198+0000 7f2351ffb700 1 -- 172.21.15.139:0/1970741784 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f234800f040 con 0x7f23540f6840 2021-11-20T06:06:19.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.198+0000 7f235b98c700 1 -- 172.21.15.139:0/1970741784 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f235410b530 con 0x7f23540f6840 2021-11-20T06:06:19.190 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.198+0000 7f2351ffb700 1 -- 172.21.15.139:0/1970741784 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23480147e0 con 0x7f23540f6840 2021-11-20T06:06:19.191 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.198+0000 7f235b98c700 1 -- 172.21.15.139:0/1970741784 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f235410ba70 con 0x7f23540f6840 2021-11-20T06:06:19.191 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.199+0000 7f235b98c700 1 -- 172.21.15.139:0/1970741784 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2354027a00 con 0x7f23540f6840 2021-11-20T06:06:19.191 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.199+0000 7f2351ffb700 1 -- 172.21.15.139:0/1970741784 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2348014a50 con 0x7f23540f6840 2021-11-20T06:06:19.194 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.199+0000 7f2351ffb700 1 --2- 172.21.15.139:0/1970741784 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f233c05c2a0 0x7f233c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:19.194 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.199+0000 7f2351ffb700 1 -- 172.21.15.139:0/1970741784 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f23480768e0 con 0x7f23540f6840 2021-11-20T06:06:19.194 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.200+0000 7f2359728700 1 --2- 172.21.15.139:0/1970741784 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f233c05c2a0 0x7f233c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:19.195 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.200+0000 7f2359728700 1 --2- 172.21.15.139:0/1970741784 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f233c05c2a0 0x7f233c05e760 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f234400b3b0 tx=0x7f2344004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:19.195 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.203+0000 7f2351ffb700 1 -- 172.21.15.139:0/1970741784 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23480492a0 con 0x7f23540f6840 2021-11-20T06:06:19.375 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.390+0000 7f235b98c700 1 -- 172.21.15.139:0/1970741784 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23540fdd30 con 0x7f23540f6840 2021-11-20T06:06:19.376 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.390+0000 7f2351ffb700 1 -- 172.21.15.139:0/1970741784 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2943 (secure 0 0 0) 0x7f234804c8c0 con 0x7f23540f6840 2021-11-20T06:06:19.377 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:19.377 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":56,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:07.358398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14564":{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:19.378 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.392+0000 7f235b98c700 1 -- 172.21.15.139:0/1970741784 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f233c05c2a0 msgr2=0x7f233c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:19.379 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.392+0000 7f235b98c700 1 --2- 172.21.15.139:0/1970741784 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f233c05c2a0 0x7f233c05e760 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f234400b3b0 tx=0x7f2344004d10).stop 2021-11-20T06:06:19.379 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.392+0000 7f235b98c700 1 -- 172.21.15.139:0/1970741784 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23540f6840 msgr2=0x7f23541063a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:19.379 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.392+0000 7f235b98c700 1 --2- 172.21.15.139:0/1970741784 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23540f6840 0x7f23541063a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f23480026d0 tx=0x7f2348009de0).stop 2021-11-20T06:06:19.379 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.392+0000 7f235b98c700 1 -- 172.21.15.139:0/1970741784 shutdown_connections 2021-11-20T06:06:19.380 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.392+0000 7f235b98c700 1 --2- 172.21.15.139:0/1970741784 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f233c05c2a0 0x7f233c05e760 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:19.380 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.392+0000 7f235b98c700 1 --2- 172.21.15.139:0/1970741784 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23540f55a0 0x7f2354105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:19.380 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.392+0000 7f235b98c700 1 --2- 172.21.15.139:0/1970741784 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f23540f6840 0x7f23541063a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:19.380 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.392+0000 7f235b98c700 1 -- 172.21.15.139:0/1970741784 >> 172.21.15.139:0/1970741784 conn(0x7f23540f0ce0 msgr2=0x7f23540fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:19.380 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.392+0000 7f235b98c700 1 -- 172.21.15.139:0/1970741784 shutdown_connections 2021-11-20T06:06:19.381 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:19.392+0000 7f235b98c700 1 -- 172.21.15.139:0/1970741784 wait complete. 2021-11-20T06:06:19.381 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 57 2021-11-20T06:06:19.388 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:19 smithi139 conmon[33501]: cluster 2021-11-20T06:06:18. 2021-11-20T06:06:19.389 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:19 smithi139 conmon[33501]: 255356+0000 mgr.smithi139.dplyxn (mgr.14162) 275 : cluster [DBG] pgmap v245: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T06:06:19.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:07.358398+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14564}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14564': {'gid': 14564, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 54, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.139:6827/3938771458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 3938771458}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 3938771458}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:19.390 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14564:up:rejoin 2021-11-20T06:06:19.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:19 smithi178 conmon[36600]: cluster 2021-11-20T06:06:18.255356+0000 mgr.smithi139.dplyxn (mgr.14162) 275 2021-11-20T06:06:19.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:19 smithi178 conmon[36600]: : cluster [DBG] pgmap v245: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2021-11-20T06:06:20.392 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:20.517 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.524+0000 7fd2ceb95700 1 -- 172.21.15.139:0/65034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2c80f6770 msgr2=0x7fd2c80f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:20.518 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.524+0000 7fd2ceb95700 1 --2- 172.21.15.139:0/65034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2c80f6770 0x7fd2c80f6bd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fd2b8004660 tx=0x7fd2b8009b30).stop 2021-11-20T06:06:20.518 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.524+0000 7fd2ceb95700 1 -- 172.21.15.139:0/65034 shutdown_connections 2021-11-20T06:06:20.519 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.524+0000 7fd2ceb95700 1 --2- 172.21.15.139:0/65034 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2c80f6770 0x7fd2c80f6bd0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:20.519 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.524+0000 7fd2ceb95700 1 --2- 172.21.15.139:0/65034 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2c80f54d0 0x7fd2c80f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:20.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.524+0000 7fd2ceb95700 1 -- 172.21.15.139:0/65034 >> 172.21.15.139:0/65034 conn(0x7fd2c80f0c30 msgr2=0x7fd2c80f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:20.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.524+0000 7fd2ceb95700 1 -- 172.21.15.139:0/65034 shutdown_connections 2021-11-20T06:06:20.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.524+0000 7fd2ceb95700 1 -- 172.21.15.139:0/65034 wait complete. 2021-11-20T06:06:20.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.525+0000 7fd2ceb95700 1 Processor -- start 2021-11-20T06:06:20.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.525+0000 7fd2ceb95700 1 -- start start 2021-11-20T06:06:20.521 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.525+0000 7fd2ceb95700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2c80f54d0 0x7fd2c8109a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:20.521 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.525+0000 7fd2ceb95700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2c80f6770 0x7fd2c810bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:20.521 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.525+0000 7fd2ceb95700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c8060520 con 0x7fd2c80f6770 2021-11-20T06:06:20.521 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.525+0000 7fd2ceb95700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c8060660 con 0x7fd2c80f54d0 2021-11-20T06:06:20.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.525+0000 7fd2cd392700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2c80f6770 0x7fd2c810bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:20.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.525+0000 7fd2cd392700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2c80f6770 0x7fd2c810bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58214/0 (socket says 172.21.15.139:58214) 2021-11-20T06:06:20.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.525+0000 7fd2cd392700 1 -- 172.21.15.139:0/1973501812 learned_addr learned my addr 172.21.15.139:0/1973501812 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:20.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.525+0000 7fd2cdb93700 1 --2- 172.21.15.139:0/1973501812 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2c80f54d0 0x7fd2c8109a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:20.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.525+0000 7fd2cd392700 1 -- 172.21.15.139:0/1973501812 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2c80f54d0 msgr2=0x7fd2c8109a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:20.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.525+0000 7fd2cd392700 1 --2- 172.21.15.139:0/1973501812 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2c80f54d0 0x7fd2c8109a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:20.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.525+0000 7fd2cd392700 1 -- 172.21.15.139:0/1973501812 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b8005040 con 0x7fd2c80f6770 2021-11-20T06:06:20.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.525+0000 7fd2cdb93700 1 --2- 172.21.15.139:0/1973501812 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2c80f54d0 0x7fd2c8109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:06:20.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.525+0000 7fd2cd392700 1 --2- 172.21.15.139:0/1973501812 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2c80f6770 0x7fd2c810bf70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fd2b8009f50 tx=0x7fd2b8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:20.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.526+0000 7fd2be7fc700 1 -- 172.21.15.139:0/1973501812 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2b8018070 con 0x7fd2c80f6770 2021-11-20T06:06:20.524 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.526+0000 7fd2be7fc700 1 -- 172.21.15.139:0/1973501812 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2b8013410 con 0x7fd2c80f6770 2021-11-20T06:06:20.524 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.526+0000 7fd2ceb95700 1 -- 172.21.15.139:0/1973501812 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c810c4c0 con 0x7fd2c80f6770 2021-11-20T06:06:20.524 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.526+0000 7fd2be7fc700 1 -- 172.21.15.139:0/1973501812 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2b801c3c0 con 0x7fd2c80f6770 2021-11-20T06:06:20.524 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.526+0000 7fd2ceb95700 1 -- 172.21.15.139:0/1973501812 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c810c9f0 con 0x7fd2c80f6770 2021-11-20T06:06:20.524 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.527+0000 7fd2ceb95700 1 -- 172.21.15.139:0/1973501812 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c80f48c0 con 0x7fd2c80f6770 2021-11-20T06:06:20.525 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.527+0000 7fd2be7fc700 1 -- 172.21.15.139:0/1973501812 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd2b801d420 con 0x7fd2c80f6770 2021-11-20T06:06:20.525 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.527+0000 7fd2be7fc700 1 --2- 172.21.15.139:0/1973501812 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd2b405c300 0x7fd2b405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:20.525 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.527+0000 7fd2be7fc700 1 -- 172.21.15.139:0/1973501812 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fd2b8077de0 con 0x7fd2c80f6770 2021-11-20T06:06:20.525 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.527+0000 7fd2cdb93700 1 --2- 172.21.15.139:0/1973501812 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd2b405c300 0x7fd2b405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:20.526 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.528+0000 7fd2cdb93700 1 --2- 172.21.15.139:0/1973501812 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd2b405c300 0x7fd2b405e7c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd2c4009fd0 tx=0x7fd2c400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:20.526 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.531+0000 7fd2be7fc700 1 -- 172.21.15.139:0/1973501812 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2b804a8d0 con 0x7fd2c80f6770 2021-11-20T06:06:20.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:20 smithi139 conmon[33501]: audit 2021-11-20T06:06:19.392137+0000 mon.smithi139 (mon.0) 703 : audit 2021-11-20T06:06:20.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:20 smithi139 conmon[33501]: [DBG] from='client.? 172.21.15.139:0/1970741784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:20.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:20 smithi178 conmon[36600]: audit 2021-11-20T06:06:19.392137+0000 mon.smithi139 (mon.0 2021-11-20T06:06:20.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:20 smithi178 conmon[36600]: ) 703 : audit [DBG] from='client.? 172.21.15.139:0/1970741784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:20.704 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.717+0000 7fd2ceb95700 1 -- 172.21.15.139:0/1973501812 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2c8045ad0 con 0x7fd2c80f6770 2021-11-20T06:06:20.705 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.718+0000 7fd2be7fc700 1 -- 172.21.15.139:0/1973501812 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2943 (secure 0 0 0) 0x7fd2b804def0 con 0x7fd2c80f6770 2021-11-20T06:06:20.706 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:20.706 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":56,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:07.358398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14564":{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:20.709 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.720+0000 7fd2ceb95700 1 -- 172.21.15.139:0/1973501812 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd2b405c300 msgr2=0x7fd2b405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:20.709 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.720+0000 7fd2ceb95700 1 --2- 172.21.15.139:0/1973501812 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd2b405c300 0x7fd2b405e7c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd2c4009fd0 tx=0x7fd2c400b040).stop 2021-11-20T06:06:20.710 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.720+0000 7fd2ceb95700 1 -- 172.21.15.139:0/1973501812 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2c80f6770 msgr2=0x7fd2c810bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:20.710 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.720+0000 7fd2ceb95700 1 --2- 172.21.15.139:0/1973501812 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2c80f6770 0x7fd2c810bf70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fd2b8009f50 tx=0x7fd2b8009b30).stop 2021-11-20T06:06:20.710 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.720+0000 7fd2ceb95700 1 -- 172.21.15.139:0/1973501812 shutdown_connections 2021-11-20T06:06:20.711 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.720+0000 7fd2ceb95700 1 --2- 172.21.15.139:0/1973501812 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd2b405c300 0x7fd2b405e7c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:20.711 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.720+0000 7fd2ceb95700 1 --2- 172.21.15.139:0/1973501812 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2c80f54d0 0x7fd2c8109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:20.712 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.720+0000 7fd2ceb95700 1 --2- 172.21.15.139:0/1973501812 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2c80f6770 0x7fd2c810bf70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:20.712 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.720+0000 7fd2ceb95700 1 -- 172.21.15.139:0/1973501812 >> 172.21.15.139:0/1973501812 conn(0x7fd2c80f0c30 msgr2=0x7fd2c80fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:20.713 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.720+0000 7fd2ceb95700 1 -- 172.21.15.139:0/1973501812 shutdown_connections 2021-11-20T06:06:20.713 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:20.720+0000 7fd2ceb95700 1 -- 172.21.15.139:0/1973501812 wait complete. 2021-11-20T06:06:20.714 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 57 2021-11-20T06:06:20.719 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:07.358398+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14564}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14564': {'gid': 14564, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 54, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.139:6827/3938771458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 3938771458}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 3938771458}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:20.720 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14564:up:rejoin 2021-11-20T06:06:21.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:21 smithi139 conmon[33501]: cluster 2021-11-20T06:06:20.255777+0000 mgr.smithi139.dplyxn (mgr.14162) 276 : cluster [DBG] pgmap v246: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2021-11-20T06:06:21.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:21 smithi139 conmon[33501]: audit 2021-11-20T06:06:20 2021-11-20T06:06:21.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:21 smithi139 conmon[33501]: .719860+0000 mon.smithi139 (mon.0) 704 : audit [DBG] from='client.? 172.21.15.139:0/1973501812' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:21.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:21 smithi178 conmon[36600]: cluster 2021-11-20T06:06:20.255777+0000 mgr.smithi139.dplyxn (mgr.14162) 276 : cluster [DBG] 2021-11-20T06:06:21.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:21 smithi178 conmon[36600]: pgmap v246: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2021-11-20T06:06:21.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:21 smithi178 conmon[36600]: audit 2021-11-20T06:06:20.719860+0000 mon.smithi139 (mon.0) 704 : audit [DBG] from='client.? 172.21.15.139:0/1973501812' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:21.721 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:21.852 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.862+0000 7f53ebac8700 1 -- 172.21.15.139:0/3402623782 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53e40f6810 msgr2=0x7f53e40f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:21.852 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.862+0000 7f53ebac8700 1 --2- 172.21.15.139:0/3402623782 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53e40f6810 0x7f53e40f6c70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f53d4004660 tx=0x7f53d4009b30).stop 2021-11-20T06:06:21.852 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.862+0000 7f53ebac8700 1 -- 172.21.15.139:0/3402623782 shutdown_connections 2021-11-20T06:06:21.853 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.862+0000 7f53ebac8700 1 --2- 172.21.15.139:0/3402623782 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53e40f6810 0x7f53e40f6c70 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:21.853 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.862+0000 7f53ebac8700 1 --2- 172.21.15.139:0/3402623782 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53e40f5570 0x7f53e40f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:21.853 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.862+0000 7f53ebac8700 1 -- 172.21.15.139:0/3402623782 >> 172.21.15.139:0/3402623782 conn(0x7f53e40f0cb0 msgr2=0x7f53e40f30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:21.853 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.862+0000 7f53ebac8700 1 -- 172.21.15.139:0/3402623782 shutdown_connections 2021-11-20T06:06:21.854 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.862+0000 7f53ebac8700 1 -- 172.21.15.139:0/3402623782 wait complete. 2021-11-20T06:06:21.854 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.862+0000 7f53ebac8700 1 Processor -- start 2021-11-20T06:06:21.855 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53ebac8700 1 -- start start 2021-11-20T06:06:21.855 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53ebac8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53e40f5570 0x7f53e40604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:21.855 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53ebac8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53e40f6810 0x7f53e410bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:21.855 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53ebac8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e410c190 con 0x7f53e40f5570 2021-11-20T06:06:21.856 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53ebac8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e410c2d0 con 0x7f53e40f6810 2021-11-20T06:06:21.856 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53e9864700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53e40f5570 0x7f53e40604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:21.856 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53e9864700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53e40f5570 0x7f53e40604f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58222/0 (socket says 172.21.15.139:58222) 2021-11-20T06:06:21.856 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53e9864700 1 -- 172.21.15.139:0/3902740682 learned_addr learned my addr 172.21.15.139:0/3902740682 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:21.856 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53e9063700 1 --2- 172.21.15.139:0/3902740682 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53e40f6810 0x7f53e410bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:21.857 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53e9864700 1 -- 172.21.15.139:0/3902740682 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53e40f6810 msgr2=0x7f53e410bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:21.857 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53e9864700 1 --2- 172.21.15.139:0/3902740682 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53e40f6810 0x7f53e410bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:21.857 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53e9864700 1 -- 172.21.15.139:0/3902740682 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d4005040 con 0x7f53e40f5570 2021-11-20T06:06:21.857 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53e9063700 1 --2- 172.21.15.139:0/3902740682 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53e40f6810 0x7f53e410bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:06:21.858 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53e9864700 1 --2- 172.21.15.139:0/3902740682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53e40f5570 0x7f53e40604f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f53e00093f0 tx=0x7f53e0006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:21.858 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53da7fc700 1 -- 172.21.15.139:0/3902740682 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53e0009ea0 con 0x7f53e40f5570 2021-11-20T06:06:21.858 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53ebac8700 1 -- 172.21.15.139:0/3902740682 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e410c5b0 con 0x7f53e40f5570 2021-11-20T06:06:21.858 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53ebac8700 1 -- 172.21.15.139:0/3902740682 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e410cb70 con 0x7f53e40f5570 2021-11-20T06:06:21.858 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53da7fc700 1 -- 172.21.15.139:0/3902740682 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53e0013af0 con 0x7f53e40f5570 2021-11-20T06:06:21.859 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.863+0000 7f53da7fc700 1 -- 172.21.15.139:0/3902740682 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53e000da00 con 0x7f53e40f5570 2021-11-20T06:06:21.859 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.864+0000 7f53ebac8700 1 -- 172.21.15.139:0/3902740682 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53cc004fa0 con 0x7f53e40f5570 2021-11-20T06:06:21.859 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.865+0000 7f53da7fc700 1 -- 172.21.15.139:0/3902740682 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f53e001a060 con 0x7f53e40f5570 2021-11-20T06:06:21.859 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.865+0000 7f53da7fc700 1 --2- 172.21.15.139:0/3902740682 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f53d005c2a0 0x7f53d005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:21.859 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.865+0000 7f53da7fc700 1 -- 172.21.15.139:0/3902740682 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f53e00761e0 con 0x7f53e40f5570 2021-11-20T06:06:21.860 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.865+0000 7f53e9063700 1 --2- 172.21.15.139:0/3902740682 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f53d005c2a0 0x7f53d005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:21.860 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.865+0000 7f53e9063700 1 --2- 172.21.15.139:0/3902740682 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f53d005c2a0 0x7f53d005e760 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f53d4009d60 tx=0x7f53d4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:21.861 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:21.869+0000 7f53da7fc700 1 -- 172.21.15.139:0/3902740682 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53e0048ba0 con 0x7f53e40f5570 2021-11-20T06:06:22.038 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:22.052+0000 7f53ebac8700 1 -- 172.21.15.139:0/3902740682 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53cc005bf0 con 0x7f53e40f5570 2021-11-20T06:06:22.039 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:22.053+0000 7f53da7fc700 1 -- 172.21.15.139:0/3902740682 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2943 (secure 0 0 0) 0x7f53e0018020 con 0x7f53e40f5570 2021-11-20T06:06:22.039 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:22.040 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":57},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":56,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:07.358398+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14564":{"gid":14564,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.139:6827/3938771458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":3938771458},{"type":"v1","addr":"172.21.15.139:6827","nonce":3938771458}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:22.041 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:22.055+0000 7f53ebac8700 1 -- 172.21.15.139:0/3902740682 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f53d005c2a0 msgr2=0x7f53d005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:22.041 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:22.055+0000 7f53ebac8700 1 --2- 172.21.15.139:0/3902740682 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f53d005c2a0 0x7f53d005e760 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f53d4009d60 tx=0x7f53d4009b30).stop 2021-11-20T06:06:22.042 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:22.055+0000 7f53ebac8700 1 -- 172.21.15.139:0/3902740682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53e40f5570 msgr2=0x7f53e40604f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:22.042 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:22.055+0000 7f53ebac8700 1 --2- 172.21.15.139:0/3902740682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53e40f5570 0x7f53e40604f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f53e00093f0 tx=0x7f53e0006ec0).stop 2021-11-20T06:06:22.042 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:22.055+0000 7f53ebac8700 1 -- 172.21.15.139:0/3902740682 shutdown_connections 2021-11-20T06:06:22.042 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:22.055+0000 7f53ebac8700 1 --2- 172.21.15.139:0/3902740682 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f53d005c2a0 0x7f53d005e760 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:22.043 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:22.055+0000 7f53ebac8700 1 --2- 172.21.15.139:0/3902740682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53e40f5570 0x7f53e40604f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:22.043 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:22.055+0000 7f53ebac8700 1 --2- 172.21.15.139:0/3902740682 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53e40f6810 0x7f53e410bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:22.043 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:22.055+0000 7f53ebac8700 1 -- 172.21.15.139:0/3902740682 >> 172.21.15.139:0/3902740682 conn(0x7f53e40f0cb0 msgr2=0x7f53e40fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:22.043 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:22.055+0000 7f53ebac8700 1 -- 172.21.15.139:0/3902740682 shutdown_connections 2021-11-20T06:06:22.043 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:22.055+0000 7f53ebac8700 1 -- 172.21.15.139:0/3902740682 wait complete. 2021-11-20T06:06:22.044 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 57 2021-11-20T06:06:22.052 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:07.358398+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14564}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14564': {'gid': 14564, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 54, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.139:6827/3938771458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 3938771458}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 3938771458}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:22.053 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14564:up:rejoin 2021-11-20T06:06:22.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:22 smithi178 conmon[36600]: audit 2021-11-20T06:06:22.054788+0000 mon.smithi139 (mon.0) 705 : audit [DBG] from='client.? 172.21.15.139:0/3902740682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:22.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:22 smithi178 conmon[36600]: 2021-11-20T06:06:22.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:22 smithi178 conmon[36600]: cluster 2021-11-20T06:06:22.214729+0000 mon.smithi139 (mon.0) 706 : cluster [WRN] Replacing daemon mds.cephfs.smithi139.riczqb as rank 0 with standby daemon mds.cephfs.smithi139.oudezh 2021-11-20T06:06:22.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:22 smithi178 conmon[36600]: cluster 2021-11-20T06:06:22.214794+0000 mon.smithi139 (mon.0) 707 : cluster [INF] MDS daemon mds.cephfs.smithi139.riczqb is removed because it is dead or otherwise unavailable. 2021-11-20T06:06:22.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:22 smithi178 conmon[36600]: cluster 2021-11-20T06:06: 2021-11-20T06:06:22.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:22 smithi178 conmon[36600]: 22.218679+0000 mon.smithi139 (mon.0) 708 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2021-11-20T06:06:22.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:22 smithi178 conmon[36600]: cluster 2021-11-20T06:06:22.221894+0000 mon.smithi139 (mon.0) 709 : cluster [DBG] 2021-11-20T06:06:22.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:22 smithi178 conmon[36600]: fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:replay} 2 up:standby 2021-11-20T06:06:22.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:22 smithi139 conmon[33501]: audit 2021-11-20T06:06:22.054788+0000 mon.smithi139 (mon.0) 2021-11-20T06:06:22.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:22 smithi139 conmon[33501]: 705 : audit [DBG] from='client.? 172.21.15.139:0/3902740682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:22.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:22 smithi139 conmon[33501]: cluster 2021-11-20T06:06:22.214729+0000 mon.smithi139 (mon.0) 706 : cluster [WRN] Replacing daemon mds.cephfs.smithi139.riczqb as rank 0 with standby daemon mds.cephfs.smithi139.oudezh 2021-11-20T06:06:22.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:22 smithi139 conmon[33501]: cluster 2021-11-20T06:06 2021-11-20T06:06:22.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:22 smithi139 conmon[33501]: :22.214794+0000 mon.smithi139 (mon.0) 707 : cluster [INF] MDS daemon mds.cephfs.smithi139.riczqb is removed because it is dead or otherwise unavailable. 2021-11-20T06:06:22.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:22 smithi139 conmon[33501]: cluster 2021-11-20T06:06:22.218679 2021-11-20T06:06:22.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:22 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 708 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2021-11-20T06:06:22.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:22 smithi139 conmon[33501]: cluster 2021-11-20T06:06:22.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:22 smithi139 conmon[33501]: 2021-11-20T06:06:22.221894+0000 mon.smithi139 (mon.0) 709 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:replay} 2 up:standby 2021-11-20T06:06:22.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:22 smithi139 conmon[33501]: 2021-11-20T06:06:23.054 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:23.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.188+0000 7fb2d913f700 1 -- 172.21.15.139:0/1488799245 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2d40f7920 msgr2=0x7fb2d40f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:23.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.188+0000 7fb2d913f700 1 --2- 172.21.15.139:0/1488799245 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2d40f7920 0x7fb2d40f7d80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fb2c8004660 tx=0x7fb2c8009b30).stop 2021-11-20T06:06:23.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.188+0000 7fb2d913f700 1 -- 172.21.15.139:0/1488799245 shutdown_connections 2021-11-20T06:06:23.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.188+0000 7fb2d913f700 1 --2- 172.21.15.139:0/1488799245 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2d40f7920 0x7fb2d40f7d80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:23.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.188+0000 7fb2d913f700 1 --2- 172.21.15.139:0/1488799245 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2d40f6680 0x7fb2d40f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:23.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.188+0000 7fb2d913f700 1 -- 172.21.15.139:0/1488799245 >> 172.21.15.139:0/1488799245 conn(0x7fb2d40f1de0 msgr2=0x7fb2d40f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:23.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.188+0000 7fb2d913f700 1 -- 172.21.15.139:0/1488799245 shutdown_connections 2021-11-20T06:06:23.179 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.188+0000 7fb2d913f700 1 -- 172.21.15.139:0/1488799245 wait complete. 2021-11-20T06:06:23.179 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.188+0000 7fb2d913f700 1 Processor -- start 2021-11-20T06:06:23.179 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.188+0000 7fb2d913f700 1 -- start start 2021-11-20T06:06:23.179 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2d913f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2d40f6680 0x7fb2d4106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:23.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2d913f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2d40f7920 0x7fb2d41073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:23.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2d913f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d41079e0 con 0x7fb2d40f7920 2021-11-20T06:06:23.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2d913f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d4107b20 con 0x7fb2d40f6680 2021-11-20T06:06:23.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2d259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2d40f7920 0x7fb2d41073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:23.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2d259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2d40f7920 0x7fb2d41073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58244/0 (socket says 172.21.15.139:58244) 2021-11-20T06:06:23.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2d259c700 1 -- 172.21.15.139:0/1387046790 learned_addr learned my addr 172.21.15.139:0/1387046790 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:23.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2d2d9d700 1 --2- 172.21.15.139:0/1387046790 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2d40f6680 0x7fb2d4106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:23.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2d259c700 1 -- 172.21.15.139:0/1387046790 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2d40f6680 msgr2=0x7fb2d4106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:23.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2d259c700 1 --2- 172.21.15.139:0/1387046790 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2d40f6680 0x7fb2d4106eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:23.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2d259c700 1 -- 172.21.15.139:0/1387046790 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c8005040 con 0x7fb2d40f7920 2021-11-20T06:06:23.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2d2d9d700 1 --2- 172.21.15.139:0/1387046790 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2d40f6680 0x7fb2d4106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:06:23.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2d259c700 1 --2- 172.21.15.139:0/1387046790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2d40f7920 0x7fb2d41073a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fb2c8009830 tx=0x7fb2c8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:23.183 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2bb7fe700 1 -- 172.21.15.139:0/1387046790 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2c8018070 con 0x7fb2d40f7920 2021-11-20T06:06:23.183 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2d913f700 1 -- 172.21.15.139:0/1387046790 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d410c560 con 0x7fb2d40f7920 2021-11-20T06:06:23.183 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2d913f700 1 -- 172.21.15.139:0/1387046790 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d410cac0 con 0x7fb2d40f7920 2021-11-20T06:06:23.183 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2bb7fe700 1 -- 172.21.15.139:0/1387046790 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2c8013410 con 0x7fb2d40f7920 2021-11-20T06:06:23.184 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.189+0000 7fb2bb7fe700 1 -- 172.21.15.139:0/1387046790 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2c801c620 con 0x7fb2d40f7920 2021-11-20T06:06:23.184 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.190+0000 7fb2d913f700 1 -- 172.21.15.139:0/1387046790 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d4027a00 con 0x7fb2d40f7920 2021-11-20T06:06:23.184 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.190+0000 7fb2bb7fe700 1 -- 172.21.15.139:0/1387046790 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb2c801c8e0 con 0x7fb2d40f7920 2021-11-20T06:06:23.184 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.191+0000 7fb2bb7fe700 1 --2- 172.21.15.139:0/1387046790 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb2bc05c2a0 0x7fb2bc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:23.184 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.191+0000 7fb2bb7fe700 1 -- 172.21.15.139:0/1387046790 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fb2c80780e0 con 0x7fb2d40f7920 2021-11-20T06:06:23.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.191+0000 7fb2d2d9d700 1 --2- 172.21.15.139:0/1387046790 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb2bc05c2a0 0x7fb2bc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:23.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.191+0000 7fb2d2d9d700 1 --2- 172.21.15.139:0/1387046790 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb2bc05c2a0 0x7fb2bc05e760 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fb2d40f7780 tx=0x7fb2c4008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:23.185 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.195+0000 7fb2bb7fe700 1 -- 172.21.15.139:0/1387046790 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb2c804aa40 con 0x7fb2d40f7920 2021-11-20T06:06:23.379 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.392+0000 7fb2d913f700 1 -- 172.21.15.139:0/1387046790 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2d40feeb0 con 0x7fb2d40f7920 2021-11-20T06:06:23.379 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.393+0000 7fb2bb7fe700 1 -- 172.21.15.139:0/1387046790 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+2944 (secure 0 0 0) 0x7fb2c804e060 con 0x7fb2d40f7920 2021-11-20T06:06:23.380 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:23.380 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":59,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:23.219712+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14604":{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":58,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:23.382 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.394+0000 7fb2d913f700 1 -- 172.21.15.139:0/1387046790 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb2bc05c2a0 msgr2=0x7fb2bc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:23.383 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.394+0000 7fb2d913f700 1 --2- 172.21.15.139:0/1387046790 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb2bc05c2a0 0x7fb2bc05e760 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fb2d40f7780 tx=0x7fb2c4008040).stop 2021-11-20T06:06:23.383 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.394+0000 7fb2d913f700 1 -- 172.21.15.139:0/1387046790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2d40f7920 msgr2=0x7fb2d41073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:23.383 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.394+0000 7fb2d913f700 1 --2- 172.21.15.139:0/1387046790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2d40f7920 0x7fb2d41073a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fb2c8009830 tx=0x7fb2c8009b30).stop 2021-11-20T06:06:23.384 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.395+0000 7fb2d913f700 1 -- 172.21.15.139:0/1387046790 shutdown_connections 2021-11-20T06:06:23.384 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.395+0000 7fb2d913f700 1 --2- 172.21.15.139:0/1387046790 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb2bc05c2a0 0x7fb2bc05e760 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:23.384 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.395+0000 7fb2d913f700 1 --2- 172.21.15.139:0/1387046790 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2d40f6680 0x7fb2d4106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:23.384 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.395+0000 7fb2d913f700 1 --2- 172.21.15.139:0/1387046790 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb2d40f7920 0x7fb2d41073a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:23.384 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.395+0000 7fb2d913f700 1 -- 172.21.15.139:0/1387046790 >> 172.21.15.139:0/1387046790 conn(0x7fb2d40f1de0 msgr2=0x7fb2d40fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:23.385 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.395+0000 7fb2d913f700 1 -- 172.21.15.139:0/1387046790 shutdown_connections 2021-11-20T06:06:23.385 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:23.395+0000 7fb2d913f700 1 -- 172.21.15.139:0/1387046790 wait complete. 2021-11-20T06:06:23.385 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 59 2021-11-20T06:06:23.391 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:23 smithi139 conmon[33501]: cluster 2021-11-20T06:06:22.256060+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:06:23.391 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:23 smithi139 conmon[33501]: 277 : cluster [DBG] pgmap v248: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:06:23.391 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:23 smithi139 conmon[33501]: cluster 2021-11-20T06:06:23.223078+0000 mon.smithi139 (mon.0) 710 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2991819277,v1:172.21.15.139:6829/2991819277] up:reconnect 2021-11-20T06:06:23.392 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:23 smithi139 conmon[33501]: cluster 2021-11-20T06: 2021-11-20T06:06:23.392 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:23 smithi139 conmon[33501]: 06:23.223125+0000 mon.smithi139 (mon.0 2021-11-20T06:06:23.392 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:23 smithi139 conmon[33501]: ) 711 : cluster [DBG] mds.? [v2:172.21.15.139:6826/811251312,v1:172.21.15.139:6827/811251312] up:boot 2021-11-20T06:06:23.392 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:23 smithi139 conmon[33501]: cluster 2021 2021-11-20T06:06:23.393 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:23 smithi139 conmon[33501]: -11-20T06:06: 2021-11-20T06:06:23.393 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:23 smithi139 conmon[33501]: 23.223181+0000 mon.smithi139 (mon.0) 712 : cluster 2021-11-20T06:06:23.393 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:23 smithi139 conmon[33501]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:reconnect} 3 up:standby 2021-11-20T06:06:23.393 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:23 smithi139 conmon[33501]: audit 2021-11-20T 2021-11-20T06:06:23.393 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:23 smithi139 conmon[33501]: 06:06:23.223284+0000 mon.smithi139 ( 2021-11-20T06:06:23.394 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:23 smithi139 conmon[33501]: mon.0) 713 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.riczqb"}]: dispatch 2021-11-20T06:06:23.395 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 59, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:23.219712+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14604}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14604': {'gid': 14604, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 58, 'state': 'up:reconnect', 'state_seq': 7, 'addr': '172.21.15.139:6829/2991819277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2991819277}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2991819277}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:23.395 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14604:up:reconnect 2021-11-20T06:06:23.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:23 smithi178 conmon[36600]: cluster 2021-11-20T06:06:22.256060+0000 mgr.smithi139.dplyxn (mgr.14162) 277 : cluster 2021-11-20T06:06:23.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:23 smithi178 conmon[36600]: [DBG] pgmap v248: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:06:23.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:23 smithi178 conmon[36600]: cluster 2021-11-20T06:06:23.223078+0000 mon.smithi139 (mon.0) 710 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2991819277,v1:172.21.15.139:6829/2991819277] up:reconnect 2021-11-20T06:06:23.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:23 smithi178 conmon[36600]: cluster 2021-11-20T06 2021-11-20T06:06:23.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:23 smithi178 conmon[36600]: :06:23.223125+0000 mon.smithi139 (mon.0) 711 : cluster [DBG] mds.? [v2:172.21.15.139:6826/811251312,v1:172.21.15.139:6827/811251312] up:boot 2021-11-20T06:06:23.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:23 smithi178 conmon[36600]: cluster 2021-11-20T06:06:23.223181+0000 mon.smithi139 (mon.0) 712 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:reconnect} 3 up:standby 2021-11-20T06:06:23.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:23 smithi178 conmon[36600]: 2021-11-20T06:06:23.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:23 smithi178 conmon[36600]: audit 2021-11-20T06:06:23.223284+0000 mon.smithi139 (mon.0) 713 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi139.riczqb"}]: dispatch 2021-11-20T06:06:24.396 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:24.519 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.527+0000 7f73f359e700 1 -- 172.21.15.139:0/365413721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73f40f9790 msgr2=0x7f73f40fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:24.519 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.527+0000 7f73f359e700 1 --2- 172.21.15.139:0/365413721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73f40f9790 0x7f73f40fbb80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f73e8004660 tx=0x7f73e8009b30).stop 2021-11-20T06:06:24.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.527+0000 7f73f359e700 1 -- 172.21.15.139:0/365413721 shutdown_connections 2021-11-20T06:06:24.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.527+0000 7f73f359e700 1 --2- 172.21.15.139:0/365413721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73f40f9790 0x7f73f40fbb80 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:24.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.527+0000 7f73f359e700 1 --2- 172.21.15.139:0/365413721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73f4060410 0x7f73f40f9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:24.520 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.527+0000 7f73f359e700 1 -- 172.21.15.139:0/365413721 >> 172.21.15.139:0/365413721 conn(0x7f73f40f0c30 msgr2=0x7f73f40f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:24.521 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.527+0000 7f73f359e700 1 -- 172.21.15.139:0/365413721 shutdown_connections 2021-11-20T06:06:24.521 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.527+0000 7f73f359e700 1 -- 172.21.15.139:0/365413721 wait complete. 2021-11-20T06:06:24.521 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.528+0000 7f73f359e700 1 Processor -- start 2021-11-20T06:06:24.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.528+0000 7f73f359e700 1 -- start start 2021-11-20T06:06:24.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.528+0000 7f73f359e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73f4060410 0x7f73f40f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:24.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.528+0000 7f73f359e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73f40f9790 0x7f73f40f5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:24.522 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.528+0000 7f73f359e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f40f5670 con 0x7f73f4060410 2021-11-20T06:06:24.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.528+0000 7f73f359e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f40f57b0 con 0x7f73f40f9790 2021-11-20T06:06:24.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.528+0000 7f73f259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73f4060410 0x7f73f40f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:24.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.528+0000 7f73f259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73f4060410 0x7f73f40f6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58256/0 (socket says 172.21.15.139:58256) 2021-11-20T06:06:24.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.528+0000 7f73f259c700 1 -- 172.21.15.139:0/169720863 learned_addr learned my addr 172.21.15.139:0/169720863 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:24.523 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.528+0000 7f73f259c700 1 -- 172.21.15.139:0/169720863 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73f40f9790 msgr2=0x7f73f40f5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:24.524 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.528+0000 7f73f259c700 1 --2- 172.21.15.139:0/169720863 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73f40f9790 0x7f73f40f5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:24.524 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.528+0000 7f73f259c700 1 -- 172.21.15.139:0/169720863 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e8005040 con 0x7f73f4060410 2021-11-20T06:06:24.524 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.529+0000 7f73f259c700 1 --2- 172.21.15.139:0/169720863 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73f4060410 0x7f73f40f6c90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f73e400c790 tx=0x7f73e4007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:24.524 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.529+0000 7f73e2ffd700 1 -- 172.21.15.139:0/169720863 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73e400e680 con 0x7f73f4060410 2021-11-20T06:06:24.525 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.529+0000 7f73e2ffd700 1 -- 172.21.15.139:0/169720863 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73e400ec90 con 0x7f73f4060410 2021-11-20T06:06:24.525 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.529+0000 7f73f359e700 1 -- 172.21.15.139:0/169720863 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f40f5a90 con 0x7f73f4060410 2021-11-20T06:06:24.525 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.529+0000 7f73e2ffd700 1 -- 172.21.15.139:0/169720863 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73e4013d60 con 0x7f73f4060410 2021-11-20T06:06:24.525 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.529+0000 7f73f359e700 1 -- 172.21.15.139:0/169720863 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f410eda0 con 0x7f73f4060410 2021-11-20T06:06:24.526 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.530+0000 7f73f359e700 1 -- 172.21.15.139:0/169720863 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f4045ad0 con 0x7f73f4060410 2021-11-20T06:06:24.526 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.530+0000 7f73e2ffd700 1 -- 172.21.15.139:0/169720863 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f73e401a060 con 0x7f73f4060410 2021-11-20T06:06:24.526 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.530+0000 7f73e2ffd700 1 --2- 172.21.15.139:0/169720863 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f73dc05c300 0x7f73dc05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:24.526 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.530+0000 7f73e2ffd700 1 -- 172.21.15.139:0/169720863 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f73e4076440 con 0x7f73f4060410 2021-11-20T06:06:24.527 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.530+0000 7f73f1d9b700 1 --2- 172.21.15.139:0/169720863 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f73dc05c300 0x7f73dc05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:24.527 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.531+0000 7f73f1d9b700 1 --2- 172.21.15.139:0/169720863 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f73dc05c300 0x7f73dc05e7c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f73e8009f50 tx=0x7f73e8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:24.527 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.534+0000 7f73e2ffd700 1 -- 172.21.15.139:0/169720863 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f73e4048d40 con 0x7f73f4060410 2021-11-20T06:06:24.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:24 smithi139 conmon[33501]: cluster 2021-11-20T06:06:23.243078+0000 mds.cephfs.smithi139.oudezh (mds.0) 1 : cluster 2021-11-20T06:06:24.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:24 smithi139 conmon[33501]: [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0219995 2021-11-20T06:06:24.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:24 smithi139 conmon[33501]: audit 2021-11-20T06:06:23.394832+0000 mon.smithi139 (mon.0) 714 : audit [DBG] from='client.? 172.21.15.139:0/1387046790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:24.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:24 smithi139 conmon[33501]: cluster 2021-11-20T06:06:24.232011+0000 mon.smithi139 (mon.0) 715 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2991819277,v1:172.21.15.139:6829/2991819277] up:rejoin 2021-11-20T06:06:24.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:24 smithi139 conmon[33501]: cluster 2021-11-20T06:06:24.232122+0000 mon.smithi139 (mon.0) 716 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:06:24.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:24 smithi178 conmon[36600]: cluster 2021-11-20T06:06:23.243078+0000 mds.cephfs.smithi139.oudezh (mds.0) 2021-11-20T06:06:24.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:24 smithi178 conmon[36600]: 1 : cluster [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0219995 2021-11-20T06:06:24.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:24 smithi178 conmon[36600]: audit 2021-11-20T06:06:23.394832+0000 mon.smithi139 (mon.0) 714 : audit [DBG] from='client.? 172.21.15.139:0/1387046790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:24.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:24 smithi178 conmon[36600]: cluster 2021-11 2021-11-20T06:06:24.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:24 smithi178 conmon[36600]: -20T06:06:24.232011+0000 mon.smithi139 (mon.0) 715 : cluster [DBG] mds.? [v2:172.21.15.139:6828/2991819277,v1:172.21.15.139:6829/2991819277] up:rejoin 2021-11-20T06:06:24.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:24 smithi178 conmon[36600]: cluster 2021-11-20T06:06:24.232122+0000 mon.smithi139 ( 2021-11-20T06:06:24.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:24 smithi178 conmon[36600]: mon.0) 716 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:06:24.710 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.725+0000 7f73f359e700 1 -- 172.21.15.139:0/169720863 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73f40fed90 con 0x7f73f4060410 2021-11-20T06:06:24.711 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.725+0000 7f73e2ffd700 1 -- 172.21.15.139:0/169720863 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+2941 (secure 0 0 0) 0x7f73e404c360 con 0x7f73f4060410 2021-11-20T06:06:24.711 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:24.712 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:24.227400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14604":{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:24.713 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.727+0000 7f73f359e700 1 -- 172.21.15.139:0/169720863 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f73dc05c300 msgr2=0x7f73dc05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:24.713 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.727+0000 7f73f359e700 1 --2- 172.21.15.139:0/169720863 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f73dc05c300 0x7f73dc05e7c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f73e8009f50 tx=0x7f73e8009b30).stop 2021-11-20T06:06:24.714 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.727+0000 7f73f359e700 1 -- 172.21.15.139:0/169720863 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73f4060410 msgr2=0x7f73f40f6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:24.714 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.727+0000 7f73f359e700 1 --2- 172.21.15.139:0/169720863 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73f4060410 0x7f73f40f6c90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f73e400c790 tx=0x7f73e4007d70).stop 2021-11-20T06:06:24.714 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.727+0000 7f73f359e700 1 -- 172.21.15.139:0/169720863 shutdown_connections 2021-11-20T06:06:24.714 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.727+0000 7f73f359e700 1 --2- 172.21.15.139:0/169720863 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f73dc05c300 0x7f73dc05e7c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:24.715 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.727+0000 7f73f359e700 1 --2- 172.21.15.139:0/169720863 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f73f4060410 0x7f73f40f6c90 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:24.715 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.727+0000 7f73f359e700 1 --2- 172.21.15.139:0/169720863 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73f40f9790 0x7f73f40f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:24.715 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.727+0000 7f73f359e700 1 -- 172.21.15.139:0/169720863 >> 172.21.15.139:0/169720863 conn(0x7f73f40f0c30 msgr2=0x7f73f40f3020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:24.715 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.727+0000 7f73f359e700 1 -- 172.21.15.139:0/169720863 shutdown_connections 2021-11-20T06:06:24.715 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:24.727+0000 7f73f359e700 1 -- 172.21.15.139:0/169720863 wait complete. 2021-11-20T06:06:24.716 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 60 2021-11-20T06:06:24.725 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:24.227400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14604}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14604': {'gid': 14604, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 58, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6829/2991819277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2991819277}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2991819277}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:24.725 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14604:up:rejoin 2021-11-20T06:06:25.492 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:25 smithi139 conmon[33501]: cluster 2021-11-20T06:06:24.256480+0000 mgr.smithi139.dplyxn (mgr.14162) 278 : cluster [DBG] pgmap v249: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T06:06:25.492 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:25 smithi139 conmon[33501]: 2021-11-20T06:06:25.492 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:25 smithi139 conmon[33501]: audit 2021-11-20T06:06:24.727003+0000 mon.smithi139 (mon.0) 717 : audit [DBG] from='client.? 172.21.15.139:0/169720863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:25.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:25 smithi178 conmon[36600]: cluster 2021-11-20T06:06:24.256480+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:06:25.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:25 smithi178 conmon[36600]: 14162) 278 : cluster [DBG] pgmap v249: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T06:06:25.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:25 smithi178 conmon[36600]: audit 2021-11-20T06:06:24.727003+0000 mon.smithi139 (mon.0) 717 : audit [DBG] from='client.? 172.21.15.139:0/169720863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:25.727 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:25.857 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.868+0000 7fbac6f06700 1 -- 172.21.15.139:0/2793948009 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbac00f6ab0 msgr2=0x7fbac00f6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:25.858 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.868+0000 7fbac6f06700 1 --2- 172.21.15.139:0/2793948009 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbac00f6ab0 0x7fbac00f6ed0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fbabc004660 tx=0x7fbabc009b30).stop 2021-11-20T06:06:25.858 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.868+0000 7fbac6f06700 1 -- 172.21.15.139:0/2793948009 shutdown_connections 2021-11-20T06:06:25.858 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.868+0000 7fbac6f06700 1 --2- 172.21.15.139:0/2793948009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbac00f7450 0x7fbac00f5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:25.858 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.868+0000 7fbac6f06700 1 --2- 172.21.15.139:0/2793948009 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbac00f6ab0 0x7fbac00f6ed0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:25.859 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.868+0000 7fbac6f06700 1 -- 172.21.15.139:0/2793948009 >> 172.21.15.139:0/2793948009 conn(0x7fbac00f0360 msgr2=0x7fbac00f27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:25.859 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.869+0000 7fbac6f06700 1 -- 172.21.15.139:0/2793948009 shutdown_connections 2021-11-20T06:06:25.859 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.869+0000 7fbac6f06700 1 -- 172.21.15.139:0/2793948009 wait complete. 2021-11-20T06:06:25.859 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.869+0000 7fbac6f06700 1 Processor -- start 2021-11-20T06:06:25.860 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.869+0000 7fbac6f06700 1 -- start start 2021-11-20T06:06:25.860 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.869+0000 7fbac6f06700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbac00f6ab0 0x7fbac0106db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:25.860 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.869+0000 7fbac6f06700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbac00f7450 0x7fbac01072a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:25.860 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.869+0000 7fbac6f06700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac01078a0 con 0x7fbac00f6ab0 2021-11-20T06:06:25.861 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.869+0000 7fbac6f06700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac01080e0 con 0x7fbac00f7450 2021-11-20T06:06:25.861 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.869+0000 7fbac5f04700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbac00f6ab0 0x7fbac0106db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:25.861 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.869+0000 7fbac5f04700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbac00f6ab0 0x7fbac0106db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58268/0 (socket says 172.21.15.139:58268) 2021-11-20T06:06:25.861 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.869+0000 7fbac5f04700 1 -- 172.21.15.139:0/2678726748 learned_addr learned my addr 172.21.15.139:0/2678726748 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:25.862 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.869+0000 7fbac5703700 1 --2- 172.21.15.139:0/2678726748 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbac00f7450 0x7fbac01072a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:25.862 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.870+0000 7fbac5f04700 1 -- 172.21.15.139:0/2678726748 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbac00f7450 msgr2=0x7fbac01072a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:25.862 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.870+0000 7fbac5f04700 1 --2- 172.21.15.139:0/2678726748 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbac00f7450 0x7fbac01072a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:25.862 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.870+0000 7fbac5f04700 1 -- 172.21.15.139:0/2678726748 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbabc005040 con 0x7fbac00f6ab0 2021-11-20T06:06:25.862 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.870+0000 7fbac5f04700 1 --2- 172.21.15.139:0/2678726748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbac00f6ab0 0x7fbac0106db0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fbabc004e30 tx=0x7fbabc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:25.863 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.870+0000 7fbab67fc700 1 -- 172.21.15.139:0/2678726748 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbabc018070 con 0x7fbac00f6ab0 2021-11-20T06:06:25.863 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.870+0000 7fbab67fc700 1 -- 172.21.15.139:0/2678726748 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbabc013410 con 0x7fbac00f6ab0 2021-11-20T06:06:25.863 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.870+0000 7fbab67fc700 1 -- 172.21.15.139:0/2678726748 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbabc01c3c0 con 0x7fbac00f6ab0 2021-11-20T06:06:25.863 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.870+0000 7fbac6f06700 1 -- 172.21.15.139:0/2678726748 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac0108360 con 0x7fbac00f6ab0 2021-11-20T06:06:25.864 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.870+0000 7fbac6f06700 1 -- 172.21.15.139:0/2678726748 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac0108840 con 0x7fbac00f6ab0 2021-11-20T06:06:25.865 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.871+0000 7fbac6f06700 1 -- 172.21.15.139:0/2678726748 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac00f4010 con 0x7fbac00f6ab0 2021-11-20T06:06:25.865 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.871+0000 7fbab67fc700 1 -- 172.21.15.139:0/2678726748 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbabc01d420 con 0x7fbac00f6ab0 2021-11-20T06:06:25.865 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.871+0000 7fbab67fc700 1 --2- 172.21.15.139:0/2678726748 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fbaac05c300 0x7fbaac05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:25.866 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.872+0000 7fbab67fc700 1 -- 172.21.15.139:0/2678726748 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fbabc077ec0 con 0x7fbac00f6ab0 2021-11-20T06:06:25.866 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.872+0000 7fbac5703700 1 --2- 172.21.15.139:0/2678726748 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fbaac05c300 0x7fbaac05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:25.866 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.872+0000 7fbac5703700 1 --2- 172.21.15.139:0/2678726748 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fbaac05c300 0x7fbaac05e7c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fbac0106b40 tx=0x7fbab0009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:25.867 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:25.875+0000 7fbab67fc700 1 -- 172.21.15.139:0/2678726748 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbabc04a820 con 0x7fbac00f6ab0 2021-11-20T06:06:26.053 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:26.065+0000 7fbac6f06700 1 -- 172.21.15.139:0/2678726748 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbac00fede0 con 0x7fbac00f6ab0 2021-11-20T06:06:26.053 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:26.065+0000 7fbab67fc700 1 -- 172.21.15.139:0/2678726748 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+2941 (secure 0 0 0) 0x7fbabc04de40 con 0x7fbac00f6ab0 2021-11-20T06:06:26.054 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:26.054 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":59},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:24.227400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14604":{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:26.055 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:26.067+0000 7fbac6f06700 1 -- 172.21.15.139:0/2678726748 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fbaac05c300 msgr2=0x7fbaac05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:26.055 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:26.067+0000 7fbac6f06700 1 --2- 172.21.15.139:0/2678726748 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fbaac05c300 0x7fbaac05e7c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fbac0106b40 tx=0x7fbab0009be0).stop 2021-11-20T06:06:26.056 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:26.067+0000 7fbac6f06700 1 -- 172.21.15.139:0/2678726748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbac00f6ab0 msgr2=0x7fbac0106db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:26.056 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:26.067+0000 7fbac6f06700 1 --2- 172.21.15.139:0/2678726748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbac00f6ab0 0x7fbac0106db0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fbabc004e30 tx=0x7fbabc009b30).stop 2021-11-20T06:06:26.056 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:26.067+0000 7fbac6f06700 1 -- 172.21.15.139:0/2678726748 shutdown_connections 2021-11-20T06:06:26.057 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:26.067+0000 7fbac6f06700 1 --2- 172.21.15.139:0/2678726748 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fbaac05c300 0x7fbaac05e7c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:26.057 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:26.067+0000 7fbac6f06700 1 --2- 172.21.15.139:0/2678726748 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbac00f6ab0 0x7fbac0106db0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:26.057 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:26.067+0000 7fbac6f06700 1 --2- 172.21.15.139:0/2678726748 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbac00f7450 0x7fbac01072a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:26.057 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:26.067+0000 7fbac6f06700 1 -- 172.21.15.139:0/2678726748 >> 172.21.15.139:0/2678726748 conn(0x7fbac00f0360 msgr2=0x7fbac00fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:26.057 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:26.067+0000 7fbac6f06700 1 -- 172.21.15.139:0/2678726748 shutdown_connections 2021-11-20T06:06:26.058 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:26.067+0000 7fbac6f06700 1 -- 172.21.15.139:0/2678726748 wait complete. 2021-11-20T06:06:26.058 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 60 2021-11-20T06:06:26.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:24.227400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14604}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14604': {'gid': 14604, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 58, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6829/2991819277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2991819277}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2991819277}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:26.066 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14604:up:rejoin 2021-11-20T06:06:26.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:26 smithi139 conmon[33501]: audit 2021-11-20T06:06:26.067311+0000 mon.smithi139 (mon.0) 718 : audit [DBG] 2021-11-20T06:06:26.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:26 smithi139 conmon[33501]: from='client.? 172.21.15.139:0/2678726748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:26.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:26 smithi178 conmon[36600]: audit 2021-11-20T06:06:26.067311+0000 mon.smithi139 (mon. 2021-11-20T06:06:26.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:26 smithi178 conmon[36600]: 0) 718 : audit [DBG] from='client.? 172.21.15.139:0/2678726748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:27.067 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:27.195 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.207+0000 7f3d93d2a700 1 -- 172.21.15.139:0/1856997644 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d8c0f7920 msgr2=0x7f3d8c0f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:27.196 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.207+0000 7f3d93d2a700 1 --2- 172.21.15.139:0/1856997644 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d8c0f7920 0x7f3d8c0f7d80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f3d7c004660 tx=0x7f3d7c009b30).stop 2021-11-20T06:06:27.196 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.207+0000 7f3d93d2a700 1 -- 172.21.15.139:0/1856997644 shutdown_connections 2021-11-20T06:06:27.196 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.207+0000 7f3d93d2a700 1 --2- 172.21.15.139:0/1856997644 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d8c0f7920 0x7f3d8c0f7d80 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:27.196 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.207+0000 7f3d93d2a700 1 --2- 172.21.15.139:0/1856997644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d8c0f6680 0x7f3d8c0f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:27.197 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.207+0000 7f3d93d2a700 1 -- 172.21.15.139:0/1856997644 >> 172.21.15.139:0/1856997644 conn(0x7f3d8c0f1de0 msgr2=0x7f3d8c0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:27.197 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.207+0000 7f3d93d2a700 1 -- 172.21.15.139:0/1856997644 shutdown_connections 2021-11-20T06:06:27.197 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.207+0000 7f3d93d2a700 1 -- 172.21.15.139:0/1856997644 wait complete. 2021-11-20T06:06:27.197 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d93d2a700 1 Processor -- start 2021-11-20T06:06:27.197 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d93d2a700 1 -- start start 2021-11-20T06:06:27.198 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d93d2a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d8c0f6680 0x7f3d8c06b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:27.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d93d2a700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d8c0f7920 0x7f3d8c069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:27.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d93d2a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c06bd70 con 0x7f3d8c0f6680 2021-11-20T06:06:27.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d93d2a700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c06a310 con 0x7f3d8c0f7920 2021-11-20T06:06:27.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d91ac6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d8c0f6680 0x7f3d8c06b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:27.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d91ac6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d8c0f6680 0x7f3d8c06b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58280/0 (socket says 172.21.15.139:58280) 2021-11-20T06:06:27.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d91ac6700 1 -- 172.21.15.139:0/1189672427 learned_addr learned my addr 172.21.15.139:0/1189672427 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:27.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d912c5700 1 --2- 172.21.15.139:0/1189672427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d8c0f7920 0x7f3d8c069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:27.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d91ac6700 1 -- 172.21.15.139:0/1189672427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d8c0f7920 msgr2=0x7f3d8c069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:27.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d91ac6700 1 --2- 172.21.15.139:0/1189672427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d8c0f7920 0x7f3d8c069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:27.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d91ac6700 1 -- 172.21.15.139:0/1189672427 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c005040 con 0x7f3d8c0f6680 2021-11-20T06:06:27.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d912c5700 1 --2- 172.21.15.139:0/1189672427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d8c0f7920 0x7f3d8c069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:06:27.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d91ac6700 1 --2- 172.21.15.139:0/1189672427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d8c0f6680 0x7f3d8c06b7c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f3d8800c710 tx=0x7f3d88004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:27.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.208+0000 7f3d827fc700 1 -- 172.21.15.139:0/1189672427 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d880106f0 con 0x7f3d8c0f6680 2021-11-20T06:06:27.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.209+0000 7f3d827fc700 1 -- 172.21.15.139:0/1189672427 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d88010d00 con 0x7f3d8c0f6680 2021-11-20T06:06:27.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.209+0000 7f3d827fc700 1 -- 172.21.15.139:0/1189672427 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d88012d90 con 0x7f3d8c0f6680 2021-11-20T06:06:27.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.209+0000 7f3d93d2a700 1 -- 172.21.15.139:0/1189672427 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8c06a590 con 0x7f3d8c0f6680 2021-11-20T06:06:27.203 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.209+0000 7f3d93d2a700 1 -- 172.21.15.139:0/1189672427 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8c06ab00 con 0x7f3d8c0f6680 2021-11-20T06:06:27.203 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.209+0000 7f3d93d2a700 1 -- 172.21.15.139:0/1189672427 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8c027a00 con 0x7f3d8c0f6680 2021-11-20T06:06:27.203 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.210+0000 7f3d827fc700 1 -- 172.21.15.139:0/1189672427 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3d88010070 con 0x7f3d8c0f6680 2021-11-20T06:06:27.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.210+0000 7f3d827fc700 1 --2- 172.21.15.139:0/1189672427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3d7805c2a0 0x7f3d7805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:27.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.210+0000 7f3d912c5700 1 --2- 172.21.15.139:0/1189672427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3d7805c2a0 0x7f3d7805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:27.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.210+0000 7f3d827fc700 1 -- 172.21.15.139:0/1189672427 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f3d88076730 con 0x7f3d8c0f6680 2021-11-20T06:06:27.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.211+0000 7f3d912c5700 1 --2- 172.21.15.139:0/1189672427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3d7805c2a0 0x7f3d7805e760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f3d7c009d60 tx=0x7f3d7c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:27.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.214+0000 7f3d827fc700 1 -- 172.21.15.139:0/1189672427 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d88049090 con 0x7f3d8c0f6680 2021-11-20T06:06:27.390 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.405+0000 7f3d93d2a700 1 -- 172.21.15.139:0/1189672427 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d8c0fee10 con 0x7f3d8c0f6680 2021-11-20T06:06:27.391 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.405+0000 7f3d827fc700 1 -- 172.21.15.139:0/1189672427 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2940 (secure 0 0 0) 0x7f3d8804c6b0 con 0x7f3d8c0f6680 2021-11-20T06:06:27.392 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:27.392 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:24.227400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14604":{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:27.394 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.407+0000 7f3d93d2a700 1 -- 172.21.15.139:0/1189672427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3d7805c2a0 msgr2=0x7f3d7805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:27.394 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.407+0000 7f3d93d2a700 1 --2- 172.21.15.139:0/1189672427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3d7805c2a0 0x7f3d7805e760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f3d7c009d60 tx=0x7f3d7c009b30).stop 2021-11-20T06:06:27.394 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.407+0000 7f3d93d2a700 1 -- 172.21.15.139:0/1189672427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d8c0f6680 msgr2=0x7f3d8c06b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:27.394 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.407+0000 7f3d93d2a700 1 --2- 172.21.15.139:0/1189672427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d8c0f6680 0x7f3d8c06b7c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f3d8800c710 tx=0x7f3d88004d10).stop 2021-11-20T06:06:27.395 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.407+0000 7f3d93d2a700 1 -- 172.21.15.139:0/1189672427 shutdown_connections 2021-11-20T06:06:27.395 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.407+0000 7f3d93d2a700 1 --2- 172.21.15.139:0/1189672427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3d7805c2a0 0x7f3d7805e760 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:27.395 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.407+0000 7f3d93d2a700 1 --2- 172.21.15.139:0/1189672427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d8c0f6680 0x7f3d8c06b7c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:27.395 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.407+0000 7f3d93d2a700 1 --2- 172.21.15.139:0/1189672427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d8c0f7920 0x7f3d8c069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:27.396 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.407+0000 7f3d93d2a700 1 -- 172.21.15.139:0/1189672427 >> 172.21.15.139:0/1189672427 conn(0x7f3d8c0f1de0 msgr2=0x7f3d8c0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:27.396 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.407+0000 7f3d93d2a700 1 -- 172.21.15.139:0/1189672427 shutdown_connections 2021-11-20T06:06:27.396 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:27.407+0000 7f3d93d2a700 1 -- 172.21.15.139:0/1189672427 wait complete. 2021-11-20T06:06:27.396 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 61 2021-11-20T06:06:27.403 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:27 smithi139 conmon[33501]: cluster 2021-11-20T06:06:26.256967+0000 mgr.smithi139.dplyxn (mgr.14162) 279 : cluster [DBG] pgmap v250: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2021-11-20T06:06:27.404 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:27 smithi139 conmon[33501]: cluster 2021-11-20T06:06:26.273569+0000 mon.smithi139 (mon.0) 719 : cluster [DBG] mds.? [v2:172.21.15.139:6826/811251312,v1:172.21.15.139:6827/811251312] up:standby 2021-11-20T06:06:27.404 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:27 smithi139 conmon[33501]: cluster 2021-11-20T06:06:26.273628+0000 mon.smithi139 (mon.0) 720 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:06:27.406 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:24.227400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14604}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14604': {'gid': 14604, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 58, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6829/2991819277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2991819277}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2991819277}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:27.406 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14604:up:rejoin 2021-11-20T06:06:27.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:27 smithi178 conmon[36600]: cluster 2021-11-20T06:06:26.256967+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:06:27.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:27 smithi178 conmon[36600]: mgr.14162) 279 : cluster [DBG] pgmap v250: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2021-11-20T06:06:27.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:27 smithi178 conmon[36600]: cluster 2021-11-20T06:06:26.273569+0000 mon.smithi139 (mon. 2021-11-20T06:06:27.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:27 smithi178 conmon[36600]: 0) 719 : cluster [DBG] mds.? [v2:172.21.15.139:6826/811251312,v1:172.21.15.139:6827/811251312] up:standby 2021-11-20T06:06:27.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:27 smithi178 conmon[36600]: cluster 2021-11-20T06:06:27.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:27 smithi178 conmon[36600]: 2021-11-20T06:06:26.273628+0000 mon.smithi139 (mon.0) 720 : cluster [DBG] 2021-11-20T06:06:27.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:27 smithi178 conmon[36600]: fsmap cephfs:1/1 {0=cephfs.smithi139.oudezh=up:rejoin} 3 up:standby 2021-11-20T06:06:28.407 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:28.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:28 smithi139 conmon[33501]: audit 2021-11-20T06:06:27.407040+0000 mon.smithi139 (mon.0) 721 : audit 2021-11-20T06:06:28.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:28 smithi139 conmon[33501]: [DBG] from='client.? 172.21.15.139:0/1189672427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:28.535 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.547+0000 7f681a389700 1 -- 172.21.15.139:0/3438104231 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68140f6850 msgr2=0x7f68140f6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:28.535 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.547+0000 7f681a389700 1 --2- 172.21.15.139:0/3438104231 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68140f6850 0x7f68140f6cb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f6808004660 tx=0x7f6808009b30).stop 2021-11-20T06:06:28.536 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.547+0000 7f681a389700 1 -- 172.21.15.139:0/3438104231 shutdown_connections 2021-11-20T06:06:28.536 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.547+0000 7f681a389700 1 --2- 172.21.15.139:0/3438104231 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68140f6850 0x7f68140f6cb0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:28.536 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.547+0000 7f681a389700 1 --2- 172.21.15.139:0/3438104231 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f68140f55b0 0x7f68140f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:28.536 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.547+0000 7f681a389700 1 -- 172.21.15.139:0/3438104231 >> 172.21.15.139:0/3438104231 conn(0x7f68140f0d10 msgr2=0x7f68140f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:28.536 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.547+0000 7f681a389700 1 -- 172.21.15.139:0/3438104231 shutdown_connections 2021-11-20T06:06:28.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.547+0000 7f681a389700 1 -- 172.21.15.139:0/3438104231 wait complete. 2021-11-20T06:06:28.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.548+0000 7f681a389700 1 Processor -- start 2021-11-20T06:06:28.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.548+0000 7f681a389700 1 -- start start 2021-11-20T06:06:28.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.548+0000 7f681a389700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f68140f55b0 0x7f68140604d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:28.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.548+0000 7f681a389700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68140f6850 0x7f681410bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:28.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.548+0000 7f681a389700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681410c100 con 0x7f68140f6850 2021-11-20T06:06:28.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.548+0000 7f681a389700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681410c240 con 0x7f68140f55b0 2021-11-20T06:06:28.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.548+0000 7f68137fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68140f6850 0x7f681410bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:28.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.548+0000 7f68137fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68140f6850 0x7f681410bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58292/0 (socket says 172.21.15.139:58292) 2021-11-20T06:06:28.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.548+0000 7f68137fe700 1 -- 172.21.15.139:0/4065623352 learned_addr learned my addr 172.21.15.139:0/4065623352 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:28.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.548+0000 7f6813fff700 1 --2- 172.21.15.139:0/4065623352 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f68140f55b0 0x7f68140604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:28.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.549+0000 7f68137fe700 1 -- 172.21.15.139:0/4065623352 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f68140f55b0 msgr2=0x7f68140604d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:28.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.549+0000 7f68137fe700 1 --2- 172.21.15.139:0/4065623352 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f68140f55b0 0x7f68140604d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:28.540 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.549+0000 7f68137fe700 1 -- 172.21.15.139:0/4065623352 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6808005040 con 0x7f68140f6850 2021-11-20T06:06:28.540 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.549+0000 7f68137fe700 1 --2- 172.21.15.139:0/4065623352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68140f6850 0x7f681410bc10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f6808009870 tx=0x7f680800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:28.540 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.549+0000 7f6810ff9700 1 -- 172.21.15.139:0/4065623352 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6808019070 con 0x7f68140f6850 2021-11-20T06:06:28.540 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.549+0000 7f681a389700 1 -- 172.21.15.139:0/4065623352 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681410c440 con 0x7f68140f6850 2021-11-20T06:06:28.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.549+0000 7f6810ff9700 1 -- 172.21.15.139:0/4065623352 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6808007e70 con 0x7f68140f6850 2021-11-20T06:06:28.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.549+0000 7f6810ff9700 1 -- 172.21.15.139:0/4065623352 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6808014c90 con 0x7f68140f6850 2021-11-20T06:06:28.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.549+0000 7f681a389700 1 -- 172.21.15.139:0/4065623352 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681410c9a0 con 0x7f68140f6850 2021-11-20T06:06:28.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.550+0000 7f681a389700 1 -- 172.21.15.139:0/4065623352 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6814027a00 con 0x7f68140f6850 2021-11-20T06:06:28.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.550+0000 7f6810ff9700 1 -- 172.21.15.139:0/4065623352 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6808025070 con 0x7f68140f6850 2021-11-20T06:06:28.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.551+0000 7f6810ff9700 1 --2- 172.21.15.139:0/4065623352 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f67fc05c2a0 0x7f67fc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:28.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.551+0000 7f6810ff9700 1 -- 172.21.15.139:0/4065623352 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f68080772b0 con 0x7f68140f6850 2021-11-20T06:06:28.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.551+0000 7f6813fff700 1 --2- 172.21.15.139:0/4065623352 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f67fc05c2a0 0x7f67fc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:28.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.551+0000 7f6813fff700 1 --2- 172.21.15.139:0/4065623352 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f67fc05c2a0 0x7f67fc05e760 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f6804009fd0 tx=0x7f680400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:28.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.555+0000 7f6810ff9700 1 -- 172.21.15.139:0/4065623352 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6808049c10 con 0x7f68140f6850 2021-11-20T06:06:28.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:28 smithi178 conmon[36600]: audit 2021-11-20T06:06:27.407040+0000 mon.smithi139 (mon.0) 2021-11-20T06:06:28.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:28 smithi178 conmon[36600]: 721 : audit [DBG] from='client.? 172.21.15.139:0/1189672427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:28.730 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.743+0000 7f681a389700 1 -- 172.21.15.139:0/4065623352 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68140fdd40 con 0x7f68140f6850 2021-11-20T06:06:28.730 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.744+0000 7f6810ff9700 1 -- 172.21.15.139:0/4065623352 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2940 (secure 0 0 0) 0x7f680804d230 con 0x7f68140f6850 2021-11-20T06:06:28.731 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:28.732 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:24.227400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14604":{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:28.734 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.745+0000 7f681a389700 1 -- 172.21.15.139:0/4065623352 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f67fc05c2a0 msgr2=0x7f67fc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:28.734 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.745+0000 7f681a389700 1 --2- 172.21.15.139:0/4065623352 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f67fc05c2a0 0x7f67fc05e760 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f6804009fd0 tx=0x7f680400b040).stop 2021-11-20T06:06:28.735 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.745+0000 7f681a389700 1 -- 172.21.15.139:0/4065623352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68140f6850 msgr2=0x7f681410bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:28.735 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.745+0000 7f681a389700 1 --2- 172.21.15.139:0/4065623352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68140f6850 0x7f681410bc10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f6808009870 tx=0x7f680800b040).stop 2021-11-20T06:06:28.736 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.746+0000 7f681a389700 1 -- 172.21.15.139:0/4065623352 shutdown_connections 2021-11-20T06:06:28.736 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.746+0000 7f681a389700 1 --2- 172.21.15.139:0/4065623352 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f67fc05c2a0 0x7f67fc05e760 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:28.736 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.746+0000 7f681a389700 1 --2- 172.21.15.139:0/4065623352 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f68140f55b0 0x7f68140604d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:28.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.746+0000 7f681a389700 1 --2- 172.21.15.139:0/4065623352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68140f6850 0x7f681410bc10 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:28.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.746+0000 7f681a389700 1 -- 172.21.15.139:0/4065623352 >> 172.21.15.139:0/4065623352 conn(0x7f68140f0d10 msgr2=0x7f68140fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:28.738 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.746+0000 7f681a389700 1 -- 172.21.15.139:0/4065623352 shutdown_connections 2021-11-20T06:06:28.738 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:28.746+0000 7f681a389700 1 -- 172.21.15.139:0/4065623352 wait complete. 2021-11-20T06:06:28.738 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 61 2021-11-20T06:06:28.744 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:24.227400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14604}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14604': {'gid': 14604, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 58, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6829/2991819277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2991819277}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2991819277}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:28.745 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14604:up:rejoin 2021-11-20T06:06:29.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:29 smithi139 conmon[33501]: cluster 2021-11-20T06:06:28.257530+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:06:29.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:29 smithi139 conmon[33501]: 280 : cluster [DBG] pgmap v251: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:29.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:29 smithi139 conmon[33501]: audit 2021-11-20T06:06:28.745603+0000 mon.smithi139 (mon.0) 722 : audit [DBG] from='client.? 172.21.15.139:0/4065623352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:29.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:29 smithi178 conmon[36600]: cluster 2021-11-20T06:06:28.257530+0000 mgr.smithi139.dplyxn (mgr.14162) 280 : cluster [DBG] pgmap v251: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:29.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:29 smithi178 conmon[36600]: 2021-11-20T06:06:29.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:29 smithi178 conmon[36600]: audit 2021-11-20T06:06:28.745603+0000 mon.smithi139 (mon.0) 722 : audit [DBG] from='client.? 172.21.15.139:0/4065623352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:29.746 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:29.867 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.880+0000 7f9c79c96700 1 -- 172.21.15.139:0/722723627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c740f87b0 msgr2=0x7f9c740faba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:29.868 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.880+0000 7f9c79c96700 1 --2- 172.21.15.139:0/722723627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c740f87b0 0x7f9c740faba0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f9c68004660 tx=0x7f9c68009b30).stop 2021-11-20T06:06:29.868 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.880+0000 7f9c79c96700 1 -- 172.21.15.139:0/722723627 shutdown_connections 2021-11-20T06:06:29.868 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.880+0000 7f9c79c96700 1 --2- 172.21.15.139:0/722723627 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c740f87b0 0x7f9c740faba0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:29.868 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.880+0000 7f9c79c96700 1 --2- 172.21.15.139:0/722723627 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c740f5ed0 0x7f9c740f82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:29.869 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.880+0000 7f9c79c96700 1 -- 172.21.15.139:0/722723627 >> 172.21.15.139:0/722723627 conn(0x7f9c740efff0 msgr2=0x7f9c740f2430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:29.869 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.880+0000 7f9c79c96700 1 -- 172.21.15.139:0/722723627 shutdown_connections 2021-11-20T06:06:29.869 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.880+0000 7f9c79c96700 1 -- 172.21.15.139:0/722723627 wait complete. 2021-11-20T06:06:29.869 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.880+0000 7f9c79c96700 1 Processor -- start 2021-11-20T06:06:29.870 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.880+0000 7f9c79c96700 1 -- start start 2021-11-20T06:06:29.870 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.880+0000 7f9c79c96700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c740f5ed0 0x7f9c74105d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:29.870 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c79c96700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c740f87b0 0x7f9c74106280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:29.870 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c79c96700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c74106910 con 0x7f9c740f87b0 2021-11-20T06:06:29.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c79c96700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7410b310 con 0x7f9c740f5ed0 2021-11-20T06:06:29.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c72ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c740f87b0 0x7f9c74106280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:29.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c72ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c740f87b0 0x7f9c74106280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58304/0 (socket says 172.21.15.139:58304) 2021-11-20T06:06:29.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c72ffd700 1 -- 172.21.15.139:0/3365680666 learned_addr learned my addr 172.21.15.139:0/3365680666 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:29.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c737fe700 1 --2- 172.21.15.139:0/3365680666 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c740f5ed0 0x7f9c74105d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:29.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c72ffd700 1 -- 172.21.15.139:0/3365680666 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c740f5ed0 msgr2=0x7f9c74105d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:29.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c72ffd700 1 --2- 172.21.15.139:0/3365680666 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c740f5ed0 0x7f9c74105d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:29.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c72ffd700 1 -- 172.21.15.139:0/3365680666 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c68005040 con 0x7f9c740f87b0 2021-11-20T06:06:29.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c72ffd700 1 --2- 172.21.15.139:0/3365680666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c740f87b0 0x7f9c74106280 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f9c68009830 tx=0x7f9c68009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:29.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c5bfff700 1 -- 172.21.15.139:0/3365680666 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c68018070 con 0x7f9c740f87b0 2021-11-20T06:06:29.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c5bfff700 1 -- 172.21.15.139:0/3365680666 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c68013410 con 0x7f9c740f87b0 2021-11-20T06:06:29.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c79c96700 1 -- 172.21.15.139:0/3365680666 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7410b450 con 0x7f9c740f87b0 2021-11-20T06:06:29.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c5bfff700 1 -- 172.21.15.139:0/3365680666 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c6801c620 con 0x7f9c740f87b0 2021-11-20T06:06:29.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.881+0000 7f9c79c96700 1 -- 172.21.15.139:0/3365680666 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7410b980 con 0x7f9c740f87b0 2021-11-20T06:06:29.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.882+0000 7f9c79c96700 1 -- 172.21.15.139:0/3365680666 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c74046330 con 0x7f9c740f87b0 2021-11-20T06:06:29.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.882+0000 7f9c5bfff700 1 -- 172.21.15.139:0/3365680666 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9c6801c8e0 con 0x7f9c740f87b0 2021-11-20T06:06:29.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.883+0000 7f9c5bfff700 1 --2- 172.21.15.139:0/3365680666 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9c5c05c2a0 0x7f9c5c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:29.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.883+0000 7f9c5bfff700 1 -- 172.21.15.139:0/3365680666 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f9c680780c0 con 0x7f9c740f87b0 2021-11-20T06:06:29.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.883+0000 7f9c737fe700 1 --2- 172.21.15.139:0/3365680666 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9c5c05c2a0 0x7f9c5c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:29.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.883+0000 7f9c737fe700 1 --2- 172.21.15.139:0/3365680666 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9c5c05c2a0 0x7f9c5c05e760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f9c74107280 tx=0x7f9c64008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:29.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:29.887+0000 7f9c5bfff700 1 -- 172.21.15.139:0/3365680666 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c6804aa20 con 0x7f9c740f87b0 2021-11-20T06:06:30.058 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:30.072+0000 7f9c79c96700 1 -- 172.21.15.139:0/3365680666 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c74027a00 con 0x7f9c740f87b0 2021-11-20T06:06:30.058 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:30.073+0000 7f9c5bfff700 1 -- 172.21.15.139:0/3365680666 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2940 (secure 0 0 0) 0x7f9c6804e040 con 0x7f9c740f87b0 2021-11-20T06:06:30.059 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:30.059 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:24.227400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14604":{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:30.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:30.074+0000 7f9c79c96700 1 -- 172.21.15.139:0/3365680666 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9c5c05c2a0 msgr2=0x7f9c5c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:30.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:30.074+0000 7f9c79c96700 1 --2- 172.21.15.139:0/3365680666 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9c5c05c2a0 0x7f9c5c05e760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f9c74107280 tx=0x7f9c64008040).stop 2021-11-20T06:06:30.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:30.074+0000 7f9c79c96700 1 -- 172.21.15.139:0/3365680666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c740f87b0 msgr2=0x7f9c74106280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:30.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:30.074+0000 7f9c79c96700 1 --2- 172.21.15.139:0/3365680666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c740f87b0 0x7f9c74106280 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f9c68009830 tx=0x7f9c68009b30).stop 2021-11-20T06:06:30.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:30.074+0000 7f9c79c96700 1 -- 172.21.15.139:0/3365680666 shutdown_connections 2021-11-20T06:06:30.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:30.074+0000 7f9c79c96700 1 --2- 172.21.15.139:0/3365680666 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9c5c05c2a0 0x7f9c5c05e760 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:30.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:30.074+0000 7f9c79c96700 1 --2- 172.21.15.139:0/3365680666 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c740f5ed0 0x7f9c74105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:30.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:30.074+0000 7f9c79c96700 1 --2- 172.21.15.139:0/3365680666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c740f87b0 0x7f9c74106280 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:30.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:30.074+0000 7f9c79c96700 1 -- 172.21.15.139:0/3365680666 >> 172.21.15.139:0/3365680666 conn(0x7f9c740efff0 msgr2=0x7f9c740f2430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:30.063 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:30.074+0000 7f9c79c96700 1 -- 172.21.15.139:0/3365680666 shutdown_connections 2021-11-20T06:06:30.063 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:30.074+0000 7f9c79c96700 1 -- 172.21.15.139:0/3365680666 wait complete. 2021-11-20T06:06:30.063 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 61 2021-11-20T06:06:30.072 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:24.227400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14604}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14604': {'gid': 14604, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 58, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6829/2991819277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2991819277}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2991819277}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:30.072 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14604:up:rejoin 2021-11-20T06:06:30.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:30 smithi139 conmon[33501]: audit 2021-11-20T06:06:30.074438+0000 mon.smithi139 (mon.0) 723 : audit [DBG] from='client.? 172.21.15.139:0/3365680666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:30.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:30 smithi178 conmon[36600]: audit 2021-11-20T06:06:30.074438+0000 mon.smithi139 (mon.0) 723 : audit [DBG] from='client.? 172.21.15.139:0/3365680666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:30.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:30 smithi178 conmon[36600]: 2021-11-20T06:06:31.074 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:31.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.215+0000 7efd4f297700 1 -- 172.21.15.139:0/3416876448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd480f6680 msgr2=0x7efd480f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:31.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.215+0000 7efd4f297700 1 --2- 172.21.15.139:0/3416876448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd480f6680 0x7efd480f6aa0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7efd380099e0 tx=0x7efd38004d10).stop 2021-11-20T06:06:31.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.215+0000 7efd4f297700 1 -- 172.21.15.139:0/3416876448 shutdown_connections 2021-11-20T06:06:31.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.215+0000 7efd4f297700 1 --2- 172.21.15.139:0/3416876448 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd480f7920 0x7efd480f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:31.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.215+0000 7efd4f297700 1 --2- 172.21.15.139:0/3416876448 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd480f6680 0x7efd480f6aa0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:31.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.215+0000 7efd4f297700 1 -- 172.21.15.139:0/3416876448 >> 172.21.15.139:0/3416876448 conn(0x7efd480f1de0 msgr2=0x7efd480f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:31.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.215+0000 7efd4f297700 1 -- 172.21.15.139:0/3416876448 shutdown_connections 2021-11-20T06:06:31.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.215+0000 7efd4f297700 1 -- 172.21.15.139:0/3416876448 wait complete. 2021-11-20T06:06:31.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd4f297700 1 Processor -- start 2021-11-20T06:06:31.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd4f297700 1 -- start start 2021-11-20T06:06:31.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd4f297700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd480f6680 0x7efd4806b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:31.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd4f297700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd480f7920 0x7efd48069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:31.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd4f297700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4806bd70 con 0x7efd480f7920 2021-11-20T06:06:31.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd4f297700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4806a310 con 0x7efd480f6680 2021-11-20T06:06:31.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd4c832700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd480f7920 0x7efd48069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:31.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd4c832700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd480f7920 0x7efd48069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58314/0 (socket says 172.21.15.139:58314) 2021-11-20T06:06:31.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd4c832700 1 -- 172.21.15.139:0/2319286963 learned_addr learned my addr 172.21.15.139:0/2319286963 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:31.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd4d033700 1 --2- 172.21.15.139:0/2319286963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd480f6680 0x7efd4806b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:31.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd4c832700 1 -- 172.21.15.139:0/2319286963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd480f6680 msgr2=0x7efd4806b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:31.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd4c832700 1 --2- 172.21.15.139:0/2319286963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd480f6680 0x7efd4806b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:31.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd4c832700 1 -- 172.21.15.139:0/2319286963 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd38009640 con 0x7efd480f7920 2021-11-20T06:06:31.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd4c832700 1 --2- 172.21.15.139:0/2319286963 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd480f7920 0x7efd48069e20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7efd3c0026d0 tx=0x7efd3c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:31.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd45ffb700 1 -- 172.21.15.139:0/2319286963 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd3c009320 con 0x7efd480f7920 2021-11-20T06:06:31.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.216+0000 7efd45ffb700 1 -- 172.21.15.139:0/2319286963 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd3c00f040 con 0x7efd480f7920 2021-11-20T06:06:31.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.217+0000 7efd4f297700 1 -- 172.21.15.139:0/2319286963 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd4806a590 con 0x7efd480f7920 2021-11-20T06:06:31.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.217+0000 7efd45ffb700 1 -- 172.21.15.139:0/2319286963 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd3c0147e0 con 0x7efd480f7920 2021-11-20T06:06:31.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.217+0000 7efd4f297700 1 -- 172.21.15.139:0/2319286963 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd4806aad0 con 0x7efd480f7920 2021-11-20T06:06:31.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.218+0000 7efd4f297700 1 -- 172.21.15.139:0/2319286963 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd48046330 con 0x7efd480f7920 2021-11-20T06:06:31.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.218+0000 7efd45ffb700 1 -- 172.21.15.139:0/2319286963 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7efd3c0077b0 con 0x7efd480f7920 2021-11-20T06:06:31.211 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.218+0000 7efd45ffb700 1 --2- 172.21.15.139:0/2319286963 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7efd3005c2a0 0x7efd3005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:31.211 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.218+0000 7efd45ffb700 1 -- 172.21.15.139:0/2319286963 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7efd3c076930 con 0x7efd480f7920 2021-11-20T06:06:31.211 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.218+0000 7efd4d033700 1 --2- 172.21.15.139:0/2319286963 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7efd3005c2a0 0x7efd3005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:31.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.219+0000 7efd4d033700 1 --2- 172.21.15.139:0/2319286963 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7efd3005c2a0 0x7efd3005e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7efd3800b3b0 tx=0x7efd38004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:31.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.222+0000 7efd45ffb700 1 -- 172.21.15.139:0/2319286963 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd3c049290 con 0x7efd480f7920 2021-11-20T06:06:31.396 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.410+0000 7efd4f297700 1 -- 172.21.15.139:0/2319286963 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd48027a00 con 0x7efd480f7920 2021-11-20T06:06:31.397 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.411+0000 7efd45ffb700 1 -- 172.21.15.139:0/2319286963 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2940 (secure 0 0 0) 0x7efd48027a00 con 0x7efd480f7920 2021-11-20T06:06:31.398 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:31.398 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:24.227400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14604":{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:31.399 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.412+0000 7efd4f297700 1 -- 172.21.15.139:0/2319286963 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7efd3005c2a0 msgr2=0x7efd3005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:31.400 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.413+0000 7efd4f297700 1 --2- 172.21.15.139:0/2319286963 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7efd3005c2a0 0x7efd3005e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7efd3800b3b0 tx=0x7efd38004d10).stop 2021-11-20T06:06:31.400 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.413+0000 7efd4f297700 1 -- 172.21.15.139:0/2319286963 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd480f7920 msgr2=0x7efd48069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:31.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.413+0000 7efd4f297700 1 --2- 172.21.15.139:0/2319286963 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd480f7920 0x7efd48069e20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7efd3c0026d0 tx=0x7efd3c009de0).stop 2021-11-20T06:06:31.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.413+0000 7efd4f297700 1 -- 172.21.15.139:0/2319286963 shutdown_connections 2021-11-20T06:06:31.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.413+0000 7efd4f297700 1 --2- 172.21.15.139:0/2319286963 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7efd3005c2a0 0x7efd3005e760 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:31.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.413+0000 7efd4f297700 1 --2- 172.21.15.139:0/2319286963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd480f6680 0x7efd4806b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:31.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.413+0000 7efd4f297700 1 --2- 172.21.15.139:0/2319286963 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efd480f7920 0x7efd48069e20 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:31.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.413+0000 7efd4f297700 1 -- 172.21.15.139:0/2319286963 >> 172.21.15.139:0/2319286963 conn(0x7efd480f1de0 msgr2=0x7efd480fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:31.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.413+0000 7efd4f297700 1 -- 172.21.15.139:0/2319286963 shutdown_connections 2021-11-20T06:06:31.403 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:31.413+0000 7efd4f297700 1 -- 172.21.15.139:0/2319286963 wait complete. 2021-11-20T06:06:31.403 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 61 2021-11-20T06:06:31.409 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:31 smithi139 conmon[33501]: cluster 2021-11-20T06:06:30.258126+0000 mgr.smithi139.dplyxn (mgr.14162) 281 : cluster [DBG] pgmap v252: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:31.409 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:31 smithi139 conmon[33501]: 2021-11-20T06:06:31.410 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:31 smithi139 conmon[33501]: audit 2021-11-20T06:06:31.256535+0000 mon.smithi139 (mon.0) 724 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:06:31.410 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:31 smithi139 conmon[33501]: audit 2021-11-20T06:06:31.257495+0000 mon.smithi139 (mon.0) 725 : 2021-11-20T06:06:31.410 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:31 smithi139 conmon[33501]: audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:06:31.411 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:24.227400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14604}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14604': {'gid': 14604, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 58, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6829/2991819277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2991819277}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2991819277}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:31.412 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14604:up:rejoin 2021-11-20T06:06:31.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:31 smithi178 conmon[36600]: cluster 2021-11-20T06:06:30.258126+0000 mgr.smithi139.dplyxn (mgr.14162) 281 : cluster 2021-11-20T06:06:31.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:31 smithi178 conmon[36600]: [DBG] pgmap v252: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:31.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:31 smithi178 conmon[36600]: audit 2021-11-20T06:06:31.256535+0000 mon.smithi139 (mon.0) 724 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:06:31.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:31 smithi178 conmon[36600]: audit 2021-11-20T06:06:31.257495 2021-11-20T06:06:31.697 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:31 smithi178 conmon[36600]: +0000 mon.smithi139 (mon.0) 725 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:06:32.413 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:32.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:32 smithi178 conmon[36600]: audit 2021-11-20T06:06:31.412835+0000 mon.smithi139 (mon.0) 726 : audit [DBG] from='client.? 172.21.15.139:0/2319286963' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:32.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:32 smithi139 conmon[33501]: audit 2021-11-20T06:06:31.412835+0000 mon.smithi139 (mon.0) 726 : audit [DBG] from='client.? 172.21.15.139:0/2319286963' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:32.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.555+0000 7f05da6f6700 1 -- 172.21.15.139:0/3310100248 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d40f5970 msgr2=0x7f05d40f5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:32.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.555+0000 7f05da6f6700 1 --2- 172.21.15.139:0/3310100248 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d40f5970 0x7f05d40f5dd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f05c8004660 tx=0x7f05c8009b30).stop 2021-11-20T06:06:32.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.555+0000 7f05da6f6700 1 -- 172.21.15.139:0/3310100248 shutdown_connections 2021-11-20T06:06:32.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.555+0000 7f05da6f6700 1 --2- 172.21.15.139:0/3310100248 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d40f5970 0x7f05d40f5dd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:32.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.555+0000 7f05da6f6700 1 --2- 172.21.15.139:0/3310100248 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05d40f6c70 0x7f05d40f7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:32.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.555+0000 7f05da6f6700 1 -- 172.21.15.139:0/3310100248 >> 172.21.15.139:0/3310100248 conn(0x7f05d40f04a0 msgr2=0x7f05d40f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:32.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.555+0000 7f05da6f6700 1 -- 172.21.15.139:0/3310100248 shutdown_connections 2021-11-20T06:06:32.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.555+0000 7f05da6f6700 1 -- 172.21.15.139:0/3310100248 wait complete. 2021-11-20T06:06:32.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.556+0000 7f05da6f6700 1 Processor -- start 2021-11-20T06:06:32.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.556+0000 7f05da6f6700 1 -- start start 2021-11-20T06:06:32.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.556+0000 7f05da6f6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d40f5970 0x7f05d4105e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:32.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.556+0000 7f05da6f6700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05d40f6c70 0x7f05d4106310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:32.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.556+0000 7f05da6f6700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d41068c0 con 0x7f05d40f5970 2021-11-20T06:06:32.549 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.556+0000 7f05da6f6700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d4106a00 con 0x7f05d40f6c70 2021-11-20T06:06:32.549 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.556+0000 7f05d3fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d40f5970 0x7f05d4105e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:32.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.556+0000 7f05d3fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d40f5970 0x7f05d4105e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58320/0 (socket says 172.21.15.139:58320) 2021-11-20T06:06:32.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.556+0000 7f05d3fff700 1 -- 172.21.15.139:0/1834525395 learned_addr learned my addr 172.21.15.139:0/1834525395 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:32.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.556+0000 7f05d37fe700 1 --2- 172.21.15.139:0/1834525395 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05d40f6c70 0x7f05d4106310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:32.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.556+0000 7f05d3fff700 1 -- 172.21.15.139:0/1834525395 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05d40f6c70 msgr2=0x7f05d4106310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:32.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.556+0000 7f05d3fff700 1 --2- 172.21.15.139:0/1834525395 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05d40f6c70 0x7f05d4106310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:32.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.556+0000 7f05d3fff700 1 -- 172.21.15.139:0/1834525395 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c8005040 con 0x7f05d40f5970 2021-11-20T06:06:32.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.556+0000 7f05d37fe700 1 --2- 172.21.15.139:0/1834525395 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05d40f6c70 0x7f05d4106310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:06:32.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.556+0000 7f05d3fff700 1 --2- 172.21.15.139:0/1834525395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d40f5970 0x7f05d4105e20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f05c4002700 tx=0x7f05c4009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:32.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.557+0000 7f05d0ff9700 1 -- 172.21.15.139:0/1834525395 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05c4009320 con 0x7f05d40f5970 2021-11-20T06:06:32.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.557+0000 7f05d0ff9700 1 -- 172.21.15.139:0/1834525395 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05c400f040 con 0x7f05d40f5970 2021-11-20T06:06:32.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.557+0000 7f05da6f6700 1 -- 172.21.15.139:0/1834525395 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d410b4a0 con 0x7f05d40f5970 2021-11-20T06:06:32.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.557+0000 7f05d0ff9700 1 -- 172.21.15.139:0/1834525395 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05c4014900 con 0x7f05d40f5970 2021-11-20T06:06:32.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.557+0000 7f05da6f6700 1 -- 172.21.15.139:0/1834525395 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d410ba60 con 0x7f05d40f5970 2021-11-20T06:06:32.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.558+0000 7f05da6f6700 1 -- 172.21.15.139:0/1834525395 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d4027a00 con 0x7f05d40f5970 2021-11-20T06:06:32.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.558+0000 7f05d0ff9700 1 -- 172.21.15.139:0/1834525395 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f05c4009480 con 0x7f05d40f5970 2021-11-20T06:06:32.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.558+0000 7f05d0ff9700 1 --2- 172.21.15.139:0/1834525395 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f05bc05c2a0 0x7f05bc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:32.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.559+0000 7f05d0ff9700 1 -- 172.21.15.139:0/1834525395 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f05c40765e0 con 0x7f05d40f5970 2021-11-20T06:06:32.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.559+0000 7f05d37fe700 1 --2- 172.21.15.139:0/1834525395 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f05bc05c2a0 0x7f05bc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:32.556 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.559+0000 7f05d37fe700 1 --2- 172.21.15.139:0/1834525395 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f05bc05c2a0 0x7f05bc05e760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f05c8005010 tx=0x7f05c8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:32.556 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.562+0000 7f05d0ff9700 1 -- 172.21.15.139:0/1834525395 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05c4048f40 con 0x7f05d40f5970 2021-11-20T06:06:32.734 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.748+0000 7f05da6f6700 1 -- 172.21.15.139:0/1834525395 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05d40fde20 con 0x7f05d40f5970 2021-11-20T06:06:32.734 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.748+0000 7f05d0ff9700 1 -- 172.21.15.139:0/1834525395 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2940 (secure 0 0 0) 0x7f05c404c560 con 0x7f05d40f5970 2021-11-20T06:06:32.735 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:32.735 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:24.227400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14604":{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:32.736 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.750+0000 7f05da6f6700 1 -- 172.21.15.139:0/1834525395 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f05bc05c2a0 msgr2=0x7f05bc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:32.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.750+0000 7f05da6f6700 1 --2- 172.21.15.139:0/1834525395 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f05bc05c2a0 0x7f05bc05e760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f05c8005010 tx=0x7f05c8009b30).stop 2021-11-20T06:06:32.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.750+0000 7f05da6f6700 1 -- 172.21.15.139:0/1834525395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d40f5970 msgr2=0x7f05d4105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:32.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.750+0000 7f05da6f6700 1 --2- 172.21.15.139:0/1834525395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d40f5970 0x7f05d4105e20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f05c4002700 tx=0x7f05c4009d80).stop 2021-11-20T06:06:32.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.750+0000 7f05da6f6700 1 -- 172.21.15.139:0/1834525395 shutdown_connections 2021-11-20T06:06:32.738 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.750+0000 7f05da6f6700 1 --2- 172.21.15.139:0/1834525395 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f05bc05c2a0 0x7f05bc05e760 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:32.738 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.750+0000 7f05da6f6700 1 --2- 172.21.15.139:0/1834525395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05d40f5970 0x7f05d4105e20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:32.738 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.750+0000 7f05da6f6700 1 --2- 172.21.15.139:0/1834525395 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05d40f6c70 0x7f05d4106310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:32.739 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.750+0000 7f05da6f6700 1 -- 172.21.15.139:0/1834525395 >> 172.21.15.139:0/1834525395 conn(0x7f05d40f04a0 msgr2=0x7f05d40fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:32.739 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.750+0000 7f05da6f6700 1 -- 172.21.15.139:0/1834525395 shutdown_connections 2021-11-20T06:06:32.739 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:32.750+0000 7f05da6f6700 1 -- 172.21.15.139:0/1834525395 wait complete. 2021-11-20T06:06:32.740 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 61 2021-11-20T06:06:32.748 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:24.227400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14604}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14604': {'gid': 14604, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 58, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6829/2991819277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2991819277}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2991819277}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:32.748 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14604:up:rejoin 2021-11-20T06:06:33.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:33 smithi139 conmon[33501]: cluster 2021-11-20T06:06:32.258502+0000 mgr.smithi139.dplyxn (mgr.14162) 282 : cluster [DBG] pgmap v253: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:33.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:33 smithi139 conmon[33501]: audit 2021-11-20 2021-11-20T06:06:33.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:33 smithi139 conmon[33501]: T06:06:32.750189+0000 mon.smithi139 (mon.0) 727 : audit [DBG] from='client.? 172.21.15.139:0/1834525395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:33.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:33 smithi178 conmon[36600]: cluster 2021-11-20T06:06:32.258502+0000 mgr.smithi139.dplyxn (mgr.14162) 282 : cluster [DBG] pgmap v253: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:33.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:33 smithi178 conmon[36600]: audit 2021-11-20T06: 2021-11-20T06:06:33.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:33 smithi178 conmon[36600]: 06:32.750189+0000 mon.smithi139 (mon.0) 727 : audit [DBG] from='client.? 172.21.15.139:0/1834525395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:33.749 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:33.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.883+0000 7fdb4ff3f700 1 -- 172.21.15.139:0/1703209154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb480f5860 msgr2=0x7fdb480f5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:33.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.883+0000 7fdb4ff3f700 1 --2- 172.21.15.139:0/1703209154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb480f5860 0x7fdb480f5ce0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fdb3c004660 tx=0x7fdb3c009b30).stop 2021-11-20T06:06:33.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.883+0000 7fdb4ff3f700 1 -- 172.21.15.139:0/1703209154 shutdown_connections 2021-11-20T06:06:33.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.883+0000 7fdb4ff3f700 1 --2- 172.21.15.139:0/1703209154 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb480f5860 0x7fdb480f5ce0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:33.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.883+0000 7fdb4ff3f700 1 --2- 172.21.15.139:0/1703209154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb480f6b60 0x7fdb480f6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:33.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.883+0000 7fdb4ff3f700 1 -- 172.21.15.139:0/1703209154 >> 172.21.15.139:0/1703209154 conn(0x7fdb480f0410 msgr2=0x7fdb480f2890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:33.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.883+0000 7fdb4ff3f700 1 -- 172.21.15.139:0/1703209154 shutdown_connections 2021-11-20T06:06:33.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.883+0000 7fdb4ff3f700 1 -- 172.21.15.139:0/1703209154 wait complete. 2021-11-20T06:06:33.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.883+0000 7fdb4ff3f700 1 Processor -- start 2021-11-20T06:06:33.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.883+0000 7fdb4ff3f700 1 -- start start 2021-11-20T06:06:33.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb4ff3f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb480f5860 0x7fdb480f5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:33.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb4ff3f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb480f6b60 0x7fdb4810bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:33.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb4ff3f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4810c1c0 con 0x7fdb480f6b60 2021-11-20T06:06:33.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb4ff3f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4810c300 con 0x7fdb480f5860 2021-11-20T06:06:33.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb4d4da700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb480f6b60 0x7fdb4810bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:33.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb4d4da700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb480f6b60 0x7fdb4810bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58334/0 (socket says 172.21.15.139:58334) 2021-11-20T06:06:33.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb4d4da700 1 -- 172.21.15.139:0/86045455 learned_addr learned my addr 172.21.15.139:0/86045455 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:33.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb4d4da700 1 -- 172.21.15.139:0/86045455 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb480f5860 msgr2=0x7fdb480f5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:33.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb4dcdb700 1 --2- 172.21.15.139:0/86045455 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb480f5860 0x7fdb480f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:33.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb4d4da700 1 --2- 172.21.15.139:0/86045455 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb480f5860 0x7fdb480f5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:33.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb4d4da700 1 -- 172.21.15.139:0/86045455 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c005040 con 0x7fdb480f6b60 2021-11-20T06:06:33.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb4d4da700 1 --2- 172.21.15.139:0/86045455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb480f6b60 0x7fdb4810bcd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fdb3c007a70 tx=0x7fdb3c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:33.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb3a7fc700 1 -- 172.21.15.139:0/86045455 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb3c018070 con 0x7fdb480f6b60 2021-11-20T06:06:33.880 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb4ff3f700 1 -- 172.21.15.139:0/86045455 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb4810c580 con 0x7fdb480f6b60 2021-11-20T06:06:33.880 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb3a7fc700 1 -- 172.21.15.139:0/86045455 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb3c009700 con 0x7fdb480f6b60 2021-11-20T06:06:33.880 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb3a7fc700 1 -- 172.21.15.139:0/86045455 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb3c01c620 con 0x7fdb480f6b60 2021-11-20T06:06:33.880 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.884+0000 7fdb4ff3f700 1 -- 172.21.15.139:0/86045455 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4810cae0 con 0x7fdb480f6b60 2021-11-20T06:06:33.880 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.885+0000 7fdb4ff3f700 1 -- 172.21.15.139:0/86045455 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb48027a00 con 0x7fdb480f6b60 2021-11-20T06:06:33.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.886+0000 7fdb3a7fc700 1 -- 172.21.15.139:0/86045455 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fdb3c01d420 con 0x7fdb480f6b60 2021-11-20T06:06:33.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.886+0000 7fdb3a7fc700 1 --2- 172.21.15.139:0/86045455 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdb3405c2a0 0x7fdb3405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:33.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.886+0000 7fdb3a7fc700 1 -- 172.21.15.139:0/86045455 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fdb3c078110 con 0x7fdb480f6b60 2021-11-20T06:06:33.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.886+0000 7fdb4dcdb700 1 --2- 172.21.15.139:0/86045455 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdb3405c2a0 0x7fdb3405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:33.882 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.886+0000 7fdb4dcdb700 1 --2- 172.21.15.139:0/86045455 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdb3405c2a0 0x7fdb3405e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fdb4805f590 tx=0x7fdb44008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:33.883 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:33.890+0000 7fdb3a7fc700 1 -- 172.21.15.139:0/86045455 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb3c04aa70 con 0x7fdb480f6b60 2021-11-20T06:06:34.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:34.075+0000 7fdb4ff3f700 1 -- 172.21.15.139:0/86045455 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb480fddf0 con 0x7fdb480f6b60 2021-11-20T06:06:34.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:34.076+0000 7fdb3a7fc700 1 -- 172.21.15.139:0/86045455 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2940 (secure 0 0 0) 0x7fdb480fddf0 con 0x7fdb480f6b60 2021-11-20T06:06:34.062 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:34.062 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:24.227400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14604":{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:34.063 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:34.077+0000 7fdb4ff3f700 1 -- 172.21.15.139:0/86045455 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdb3405c2a0 msgr2=0x7fdb3405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:34.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:34.077+0000 7fdb4ff3f700 1 --2- 172.21.15.139:0/86045455 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdb3405c2a0 0x7fdb3405e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fdb4805f590 tx=0x7fdb44008040).stop 2021-11-20T06:06:34.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:34.077+0000 7fdb4ff3f700 1 -- 172.21.15.139:0/86045455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb480f6b60 msgr2=0x7fdb4810bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:34.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:34.077+0000 7fdb4ff3f700 1 --2- 172.21.15.139:0/86045455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb480f6b60 0x7fdb4810bcd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fdb3c007a70 tx=0x7fdb3c009b30).stop 2021-11-20T06:06:34.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:34.077+0000 7fdb4ff3f700 1 -- 172.21.15.139:0/86045455 shutdown_connections 2021-11-20T06:06:34.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:34.077+0000 7fdb4ff3f700 1 --2- 172.21.15.139:0/86045455 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdb3405c2a0 0x7fdb3405e760 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:34.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:34.077+0000 7fdb4ff3f700 1 --2- 172.21.15.139:0/86045455 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdb480f5860 0x7fdb480f5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:34.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:34.077+0000 7fdb4ff3f700 1 --2- 172.21.15.139:0/86045455 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdb480f6b60 0x7fdb4810bcd0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:34.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:34.077+0000 7fdb4ff3f700 1 -- 172.21.15.139:0/86045455 >> 172.21.15.139:0/86045455 conn(0x7fdb480f0410 msgr2=0x7fdb480fa2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:34.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:34.077+0000 7fdb4ff3f700 1 -- 172.21.15.139:0/86045455 shutdown_connections 2021-11-20T06:06:34.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:34.077+0000 7fdb4ff3f700 1 -- 172.21.15.139:0/86045455 wait complete. 2021-11-20T06:06:34.066 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 61 2021-11-20T06:06:34.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:24.227400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14604}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14604': {'gid': 14604, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 58, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6829/2991819277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2991819277}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2991819277}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:34.075 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14604:up:rejoin 2021-11-20T06:06:34.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:34 smithi139 conmon[33501]: audit 2021-11-20T06:06:34.077574+0000 mon.smithi139 (mon.0) 2021-11-20T06:06:34.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:34 smithi139 conmon[33501]: 728 : audit [DBG] from='client.? 172.21.15.139:0/86045455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:34.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:34 smithi178 conmon[36600]: audit 2021-11-20T06:06:34.077574+0000 mon.smithi139 (mon 2021-11-20T06:06:34.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:34 smithi178 conmon[36600]: .0) 728 : audit [DBG] from='client.? 172.21.15.139:0/86045455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:35.077 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:35.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.217+0000 7f668d745700 1 -- 172.21.15.139:0/1980079641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66880f55a0 msgr2=0x7f66880f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:35.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.217+0000 7f668d745700 1 --2- 172.21.15.139:0/1980079641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66880f55a0 0x7f66880f59c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f66780099e0 tx=0x7f6678004d10).stop 2021-11-20T06:06:35.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.217+0000 7f668d745700 1 -- 172.21.15.139:0/1980079641 shutdown_connections 2021-11-20T06:06:35.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.217+0000 7f668d745700 1 --2- 172.21.15.139:0/1980079641 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66880f6840 0x7f66880f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:35.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.217+0000 7f668d745700 1 --2- 172.21.15.139:0/1980079641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66880f55a0 0x7f66880f59c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:35.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.217+0000 7f668d745700 1 -- 172.21.15.139:0/1980079641 >> 172.21.15.139:0/1980079641 conn(0x7f66880f0ce0 msgr2=0x7f66880f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:35.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.217+0000 7f668d745700 1 -- 172.21.15.139:0/1980079641 shutdown_connections 2021-11-20T06:06:35.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.217+0000 7f668d745700 1 -- 172.21.15.139:0/1980079641 wait complete. 2021-11-20T06:06:35.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.217+0000 7f668d745700 1 Processor -- start 2021-11-20T06:06:35.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f668d745700 1 -- start start 2021-11-20T06:06:35.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f668d745700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66880f55a0 0x7f668805ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:35.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f668d745700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66880f6840 0x7f668810bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:35.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f668d745700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668810c2b0 con 0x7f66880f6840 2021-11-20T06:06:35.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f668d745700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668810c3f0 con 0x7f66880f55a0 2021-11-20T06:06:35.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f66867fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66880f6840 0x7f668810bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:35.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f66867fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66880f6840 0x7f668810bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58344/0 (socket says 172.21.15.139:58344) 2021-11-20T06:06:35.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f66867fc700 1 -- 172.21.15.139:0/3957414569 learned_addr learned my addr 172.21.15.139:0/3957414569 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:35.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f6686ffd700 1 --2- 172.21.15.139:0/3957414569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66880f55a0 0x7f668805ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:35.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f66867fc700 1 -- 172.21.15.139:0/3957414569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66880f55a0 msgr2=0x7f668805ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:35.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f66867fc700 1 --2- 172.21.15.139:0/3957414569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66880f55a0 0x7f668805ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:35.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f66867fc700 1 -- 172.21.15.139:0/3957414569 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6678009640 con 0x7f66880f6840 2021-11-20T06:06:35.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f66867fc700 1 --2- 172.21.15.139:0/3957414569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66880f6840 0x7f668810bdc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f667000c530 tx=0x7f6670007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:35.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f667f7fe700 1 -- 172.21.15.139:0/3957414569 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f667000f650 con 0x7f66880f6840 2021-11-20T06:06:35.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f667f7fe700 1 -- 172.21.15.139:0/3957414569 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f667000fc60 con 0x7f66880f6840 2021-11-20T06:06:35.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f668d745700 1 -- 172.21.15.139:0/3957414569 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668810c650 con 0x7f66880f6840 2021-11-20T06:06:35.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f667f7fe700 1 -- 172.21.15.139:0/3957414569 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6670012c30 con 0x7f66880f6840 2021-11-20T06:06:35.211 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.218+0000 7f668d745700 1 -- 172.21.15.139:0/3957414569 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f668810cb60 con 0x7f66880f6840 2021-11-20T06:06:35.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.220+0000 7f668d745700 1 -- 172.21.15.139:0/3957414569 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6688046330 con 0x7f66880f6840 2021-11-20T06:06:35.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.220+0000 7f667f7fe700 1 -- 172.21.15.139:0/3957414569 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6670010070 con 0x7f66880f6840 2021-11-20T06:06:35.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.220+0000 7f667f7fe700 1 --2- 172.21.15.139:0/3957414569 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f667405c2a0 0x7f667405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:35.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.221+0000 7f667f7fe700 1 -- 172.21.15.139:0/3957414569 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f6670075e90 con 0x7f66880f6840 2021-11-20T06:06:35.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.221+0000 7f6686ffd700 1 --2- 172.21.15.139:0/3957414569 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f667405c2a0 0x7f667405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:35.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.221+0000 7f6686ffd700 1 --2- 172.21.15.139:0/3957414569 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f667405c2a0 0x7f667405e760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f667800b620 tx=0x7f6678004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:35.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.225+0000 7f667f7fe700 1 -- 172.21.15.139:0/3957414569 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66700452b0 con 0x7f66880f6840 2021-11-20T06:06:35.395 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.409+0000 7f668d745700 1 -- 172.21.15.139:0/3957414569 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6688027a00 con 0x7f66880f6840 2021-11-20T06:06:35.396 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.410+0000 7f667f7fe700 1 -- 172.21.15.139:0/3957414569 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2940 (secure 0 0 0) 0x7f66700487b0 con 0x7f66880f6840 2021-11-20T06:06:35.396 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:35.396 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:24.227400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14604":{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:35.398 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.411+0000 7f668d745700 1 -- 172.21.15.139:0/3957414569 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f667405c2a0 msgr2=0x7f667405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:35.398 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.411+0000 7f668d745700 1 --2- 172.21.15.139:0/3957414569 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f667405c2a0 0x7f667405e760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f667800b620 tx=0x7f6678004d10).stop 2021-11-20T06:06:35.398 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.411+0000 7f668d745700 1 -- 172.21.15.139:0/3957414569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66880f6840 msgr2=0x7f668810bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:35.399 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.411+0000 7f668d745700 1 --2- 172.21.15.139:0/3957414569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66880f6840 0x7f668810bdc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f667000c530 tx=0x7f6670007e20).stop 2021-11-20T06:06:35.399 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.411+0000 7f668d745700 1 -- 172.21.15.139:0/3957414569 shutdown_connections 2021-11-20T06:06:35.399 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.411+0000 7f668d745700 1 --2- 172.21.15.139:0/3957414569 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f667405c2a0 0x7f667405e760 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:35.399 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.411+0000 7f668d745700 1 --2- 172.21.15.139:0/3957414569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66880f55a0 0x7f668805ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:35.399 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.411+0000 7f668d745700 1 --2- 172.21.15.139:0/3957414569 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f66880f6840 0x7f668810bdc0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:35.400 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.411+0000 7f668d745700 1 -- 172.21.15.139:0/3957414569 >> 172.21.15.139:0/3957414569 conn(0x7f66880f0ce0 msgr2=0x7f66880fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:35.400 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.411+0000 7f668d745700 1 -- 172.21.15.139:0/3957414569 shutdown_connections 2021-11-20T06:06:35.400 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:35.411+0000 7f668d745700 1 -- 172.21.15.139:0/3957414569 wait complete. 2021-11-20T06:06:35.401 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 61 2021-11-20T06:06:35.407 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:35 smithi139 conmon[33501]: cluster 2021-11-20T06:06:34.258916+0000 mgr.smithi139.dplyxn (mgr.14162) 283 : cluster [DBG] pgmap v254: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:06:35.408 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:24.227400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14604}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14604': {'gid': 14604, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 58, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6829/2991819277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2991819277}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2991819277}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:35.409 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14604:up:rejoin 2021-11-20T06:06:35.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:35 smithi178 conmon[36600]: cluster 2021-11-20T06:06:34.258916+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:06:35.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:35 smithi178 conmon[36600]: 14162) 283 : cluster [DBG] pgmap v254: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:06:36.410 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:36.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.549+0000 7f37129ad700 1 -- 172.21.15.139:0/1216378251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f370c0f5280 msgr2=0x7f370c0f7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:36.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.549+0000 7f37129ad700 1 --2- 172.21.15.139:0/1216378251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f370c0f5280 0x7f370c0f7bc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f36fc004660 tx=0x7f36fc009b30).stop 2021-11-20T06:06:36.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.549+0000 7f37129ad700 1 -- 172.21.15.139:0/1216378251 shutdown_connections 2021-11-20T06:06:36.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.549+0000 7f37129ad700 1 --2- 172.21.15.139:0/1216378251 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f370c0f5280 0x7f370c0f7bc0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:36.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.549+0000 7f37129ad700 1 --2- 172.21.15.139:0/1216378251 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f370c0f48e0 0x7f370c0f4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:36.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.549+0000 7f37129ad700 1 -- 172.21.15.139:0/1216378251 >> 172.21.15.139:0/1216378251 conn(0x7f370c0f0360 msgr2=0x7f370c0f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:36.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.549+0000 7f37129ad700 1 -- 172.21.15.139:0/1216378251 shutdown_connections 2021-11-20T06:06:36.540 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.549+0000 7f37129ad700 1 -- 172.21.15.139:0/1216378251 wait complete. 2021-11-20T06:06:36.540 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.550+0000 7f37129ad700 1 Processor -- start 2021-11-20T06:06:36.540 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.550+0000 7f37129ad700 1 -- start start 2021-11-20T06:06:36.540 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.550+0000 7f37129ad700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f370c0f48e0 0x7f370c106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:36.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.550+0000 7f37129ad700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f370c0f5280 0x7f370c1074a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:36.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.550+0000 7f37129ad700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370c107a50 con 0x7f370c0f48e0 2021-11-20T06:06:36.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.550+0000 7f37129ad700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370c107b90 con 0x7f370c0f5280 2021-11-20T06:06:36.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.550+0000 7f37119ab700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f370c0f48e0 0x7f370c106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:36.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.550+0000 7f37119ab700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f370c0f48e0 0x7f370c106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58350/0 (socket says 172.21.15.139:58350) 2021-11-20T06:06:36.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.550+0000 7f37119ab700 1 -- 172.21.15.139:0/1078639552 learned_addr learned my addr 172.21.15.139:0/1078639552 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:36.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.550+0000 7f37111aa700 1 --2- 172.21.15.139:0/1078639552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f370c0f5280 0x7f370c1074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:36.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.550+0000 7f37119ab700 1 -- 172.21.15.139:0/1078639552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f370c0f5280 msgr2=0x7f370c1074a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:36.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.550+0000 7f37119ab700 1 --2- 172.21.15.139:0/1078639552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f370c0f5280 0x7f370c1074a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:36.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.550+0000 7f37119ab700 1 -- 172.21.15.139:0/1078639552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36fc005040 con 0x7f370c0f48e0 2021-11-20T06:06:36.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.550+0000 7f37111aa700 1 --2- 172.21.15.139:0/1078639552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f370c0f5280 0x7f370c1074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:06:36.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.550+0000 7f37119ab700 1 --2- 172.21.15.139:0/1078639552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f370c0f48e0 0x7f370c106fb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f37080026d0 tx=0x7f3708009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:36.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.551+0000 7f37027fc700 1 -- 172.21.15.139:0/1078639552 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3708009320 con 0x7f370c0f48e0 2021-11-20T06:06:36.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.551+0000 7f37027fc700 1 -- 172.21.15.139:0/1078639552 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f370800f040 con 0x7f370c0f48e0 2021-11-20T06:06:36.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.551+0000 7f37129ad700 1 -- 172.21.15.139:0/1078639552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f370c108530 con 0x7f370c0f48e0 2021-11-20T06:06:36.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.551+0000 7f37027fc700 1 -- 172.21.15.139:0/1078639552 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37080147e0 con 0x7f370c0f48e0 2021-11-20T06:06:36.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.551+0000 7f37129ad700 1 -- 172.21.15.139:0/1078639552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f370c108af0 con 0x7f370c0f48e0 2021-11-20T06:06:36.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.552+0000 7f37129ad700 1 -- 172.21.15.139:0/1078639552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f370c045ad0 con 0x7f370c0f48e0 2021-11-20T06:06:36.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.552+0000 7f37027fc700 1 -- 172.21.15.139:0/1078639552 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3708009480 con 0x7f370c0f48e0 2021-11-20T06:06:36.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.552+0000 7f37027fc700 1 --2- 172.21.15.139:0/1078639552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f36f805c350 0x7f36f805e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:36.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.552+0000 7f37027fc700 1 -- 172.21.15.139:0/1078639552 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f37080768d0 con 0x7f370c0f48e0 2021-11-20T06:06:36.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.552+0000 7f37111aa700 1 --2- 172.21.15.139:0/1078639552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f36f805c350 0x7f36f805e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:36.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.553+0000 7f37111aa700 1 --2- 172.21.15.139:0/1078639552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f36f805c350 0x7f36f805e810 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f36fc009d60 tx=0x7f36fc009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:36.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.556+0000 7f37027fc700 1 -- 172.21.15.139:0/1078639552 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37080491f0 con 0x7f370c0f48e0 2021-11-20T06:06:36.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:36 smithi178 conmon[36600]: audit 2021-11-20T06:06:35.411505+0000 mon.smithi139 (mon.0) 729 : audit 2021-11-20T06:06:36.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:36 smithi178 conmon[36600]: [DBG] from='client.? 172.21.15.139:0/3957414569' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:36.725 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.739+0000 7f37129ad700 1 -- 172.21.15.139:0/1078639552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f370c05f5a0 con 0x7f370c0f48e0 2021-11-20T06:06:36.725 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.740+0000 7f37027fc700 1 -- 172.21.15.139:0/1078639552 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2940 (secure 0 0 0) 0x7f370804c810 con 0x7f370c0f48e0 2021-11-20T06:06:36.726 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:36.727 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:24.227400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14604":{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:36.729 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.742+0000 7f37129ad700 1 -- 172.21.15.139:0/1078639552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f36f805c350 msgr2=0x7f36f805e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:36.730 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.742+0000 7f37129ad700 1 --2- 172.21.15.139:0/1078639552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f36f805c350 0x7f36f805e810 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f36fc009d60 tx=0x7f36fc009b30).stop 2021-11-20T06:06:36.730 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.742+0000 7f37129ad700 1 -- 172.21.15.139:0/1078639552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f370c0f48e0 msgr2=0x7f370c106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:36.730 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.742+0000 7f37129ad700 1 --2- 172.21.15.139:0/1078639552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f370c0f48e0 0x7f370c106fb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f37080026d0 tx=0x7f3708009de0).stop 2021-11-20T06:06:36.731 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.742+0000 7f37129ad700 1 -- 172.21.15.139:0/1078639552 shutdown_connections 2021-11-20T06:06:36.731 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.742+0000 7f37129ad700 1 --2- 172.21.15.139:0/1078639552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f36f805c350 0x7f36f805e810 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:36.732 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.742+0000 7f37129ad700 1 --2- 172.21.15.139:0/1078639552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f370c0f48e0 0x7f370c106fb0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:36.732 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.742+0000 7f37129ad700 1 --2- 172.21.15.139:0/1078639552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f370c0f5280 0x7f370c1074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:36.733 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.742+0000 7f37129ad700 1 -- 172.21.15.139:0/1078639552 >> 172.21.15.139:0/1078639552 conn(0x7f370c0f0360 msgr2=0x7f370c0fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:36.733 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.742+0000 7f37129ad700 1 -- 172.21.15.139:0/1078639552 shutdown_connections 2021-11-20T06:06:36.733 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:36.742+0000 7f37129ad700 1 -- 172.21.15.139:0/1078639552 wait complete. 2021-11-20T06:06:36.734 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 61 2021-11-20T06:06:36.737 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:36 smithi139 conmon[33501]: audit 2021-11-20T06:06:35.411505+0000 mon.smithi139 (mon.0) 729 : audit 2021-11-20T06:06:36.738 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:36 smithi139 conmon[33501]: [DBG] from='client.? 172.21.15.139:0/3957414569' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:36.742 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:24.227400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14604}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14604': {'gid': 14604, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 58, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6829/2991819277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2991819277}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2991819277}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:36.742 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14604:up:rejoin 2021-11-20T06:06:37.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:37 smithi178 conmon[36600]: cluster 2021-11-20T06:06:36.259505+0000 mgr.smithi139.dplyxn (mgr.14162) 284 : cluster [DBG] pgmap v255: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 0 op/s 2021-11-20T06:06:37.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:37 smithi178 conmon[36600]: audit 2021-11-20T06:06:36.740793+0000 mon.smithi139 (mon.0) 730 : audit [DBG] from='client.? 172.21.15.139:0/1078639552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:37.744 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:37.772 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:37 smithi139 conmon[33501]: cluster 2021-11-20T06:06:36.259505+0000 mgr.smithi139.dplyxn (mgr.14162) 284 : cluster [DBG] pgmap v255: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 0 op/s 2021-11-20T06:06:37.772 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:37 smithi139 conmon[33501]: audit 2021-11-20T06:06:36.740793+0000 mon.smithi139 (mon.0) 730 : audit [DBG] from='client.? 172.21.15.139:0/1078639552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:37.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.885+0000 7fd4161e7700 1 -- 172.21.15.139:0/4207900248 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4100f6290 msgr2=0x7fd4100f66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:37.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.885+0000 7fd4161e7700 1 --2- 172.21.15.139:0/4207900248 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4100f6290 0x7fd4100f66f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fd400004660 tx=0x7fd400009b30).stop 2021-11-20T06:06:37.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.885+0000 7fd4161e7700 1 -- 172.21.15.139:0/4207900248 shutdown_connections 2021-11-20T06:06:37.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.885+0000 7fd4161e7700 1 --2- 172.21.15.139:0/4207900248 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4100f6290 0x7fd4100f66f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:37.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.885+0000 7fd4161e7700 1 --2- 172.21.15.139:0/4207900248 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4100f7c80 0x7fd4100f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:37.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.885+0000 7fd4161e7700 1 -- 172.21.15.139:0/4207900248 >> 172.21.15.139:0/4207900248 conn(0x7fd4100f1d10 msgr2=0x7fd4100f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:37.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.886+0000 7fd4161e7700 1 -- 172.21.15.139:0/4207900248 shutdown_connections 2021-11-20T06:06:37.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.886+0000 7fd4161e7700 1 -- 172.21.15.139:0/4207900248 wait complete. 2021-11-20T06:06:37.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.886+0000 7fd4161e7700 1 Processor -- start 2021-11-20T06:06:37.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.886+0000 7fd4161e7700 1 -- start start 2021-11-20T06:06:37.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.886+0000 7fd4161e7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4100f6290 0x7fd410109000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:37.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.886+0000 7fd4161e7700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4100f7c80 0x7fd4101094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:37.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.886+0000 7fd4161e7700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd410109aa0 con 0x7fd4100f6290 2021-11-20T06:06:37.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.886+0000 7fd4161e7700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd410109be0 con 0x7fd4100f7c80 2021-11-20T06:06:37.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.886+0000 7fd4151e5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4100f6290 0x7fd410109000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:37.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.886+0000 7fd4151e5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4100f6290 0x7fd410109000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58360/0 (socket says 172.21.15.139:58360) 2021-11-20T06:06:37.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.886+0000 7fd4151e5700 1 -- 172.21.15.139:0/1261295227 learned_addr learned my addr 172.21.15.139:0/1261295227 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:37.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.886+0000 7fd4149e4700 1 --2- 172.21.15.139:0/1261295227 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4100f7c80 0x7fd4101094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:37.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.887+0000 7fd4151e5700 1 -- 172.21.15.139:0/1261295227 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4100f7c80 msgr2=0x7fd4101094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:37.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.887+0000 7fd4151e5700 1 --2- 172.21.15.139:0/1261295227 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4100f7c80 0x7fd4101094f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:37.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.887+0000 7fd4151e5700 1 -- 172.21.15.139:0/1261295227 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd400005040 con 0x7fd4100f6290 2021-11-20T06:06:37.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.887+0000 7fd4151e5700 1 --2- 172.21.15.139:0/1261295227 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4100f6290 0x7fd410109000 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fd40c002700 tx=0x7fd40c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:37.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.887+0000 7fd405ffb700 1 -- 172.21.15.139:0/1261295227 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd40c009320 con 0x7fd4100f6290 2021-11-20T06:06:37.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.887+0000 7fd405ffb700 1 -- 172.21.15.139:0/1261295227 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd40c00f040 con 0x7fd4100f6290 2021-11-20T06:06:37.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.887+0000 7fd4161e7700 1 -- 172.21.15.139:0/1261295227 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd41010a580 con 0x7fd4100f6290 2021-11-20T06:06:37.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.887+0000 7fd405ffb700 1 -- 172.21.15.139:0/1261295227 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd40c014900 con 0x7fd4100f6290 2021-11-20T06:06:37.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.887+0000 7fd4161e7700 1 -- 172.21.15.139:0/1261295227 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd41010ab40 con 0x7fd4100f6290 2021-11-20T06:06:37.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.888+0000 7fd4161e7700 1 -- 172.21.15.139:0/1261295227 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd410045ad0 con 0x7fd4100f6290 2021-11-20T06:06:37.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.888+0000 7fd405ffb700 1 -- 172.21.15.139:0/1261295227 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd40c014a60 con 0x7fd4100f6290 2021-11-20T06:06:37.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.889+0000 7fd405ffb700 1 --2- 172.21.15.139:0/1261295227 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd3fc05c300 0x7fd3fc05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:37.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.889+0000 7fd405ffb700 1 -- 172.21.15.139:0/1261295227 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fd40c076520 con 0x7fd4100f6290 2021-11-20T06:06:37.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.889+0000 7fd4149e4700 1 --2- 172.21.15.139:0/1261295227 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd3fc05c300 0x7fd3fc05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:37.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.889+0000 7fd4149e4700 1 --2- 172.21.15.139:0/1261295227 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd3fc05c300 0x7fd3fc05e7c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fd400004e30 tx=0x7fd400009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:37.882 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:37.893+0000 7fd405ffb700 1 -- 172.21.15.139:0/1261295227 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd40c048e80 con 0x7fd4100f6290 2021-11-20T06:06:38.063 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:38.078+0000 7fd4161e7700 1 -- 172.21.15.139:0/1261295227 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd410027a00 con 0x7fd4100f6290 2021-11-20T06:06:38.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:38.079+0000 7fd405ffb700 1 -- 172.21.15.139:0/1261295227 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2940 (secure 0 0 0) 0x7fd40c04c4a0 con 0x7fd4100f6290 2021-11-20T06:06:38.064 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:38.065 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:24.227400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14604":{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:38.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:38.080+0000 7fd4161e7700 1 -- 172.21.15.139:0/1261295227 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd3fc05c300 msgr2=0x7fd3fc05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:38.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:38.080+0000 7fd4161e7700 1 --2- 172.21.15.139:0/1261295227 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd3fc05c300 0x7fd3fc05e7c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fd400004e30 tx=0x7fd400009b30).stop 2021-11-20T06:06:38.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:38.080+0000 7fd4161e7700 1 -- 172.21.15.139:0/1261295227 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4100f6290 msgr2=0x7fd410109000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:38.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:38.080+0000 7fd4161e7700 1 --2- 172.21.15.139:0/1261295227 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4100f6290 0x7fd410109000 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fd40c002700 tx=0x7fd40c009d80).stop 2021-11-20T06:06:38.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:38.080+0000 7fd4161e7700 1 -- 172.21.15.139:0/1261295227 shutdown_connections 2021-11-20T06:06:38.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:38.080+0000 7fd4161e7700 1 --2- 172.21.15.139:0/1261295227 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd3fc05c300 0x7fd3fc05e7c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:38.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:38.080+0000 7fd4161e7700 1 --2- 172.21.15.139:0/1261295227 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4100f6290 0x7fd410109000 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:38.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:38.080+0000 7fd4161e7700 1 --2- 172.21.15.139:0/1261295227 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4100f7c80 0x7fd4101094f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:38.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:38.080+0000 7fd4161e7700 1 -- 172.21.15.139:0/1261295227 >> 172.21.15.139:0/1261295227 conn(0x7fd4100f1d10 msgr2=0x7fd4100fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:38.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:38.080+0000 7fd4161e7700 1 -- 172.21.15.139:0/1261295227 shutdown_connections 2021-11-20T06:06:38.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:38.080+0000 7fd4161e7700 1 -- 172.21.15.139:0/1261295227 wait complete. 2021-11-20T06:06:38.069 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 61 2021-11-20T06:06:38.078 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:24.227400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14604}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14604': {'gid': 14604, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 58, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6829/2991819277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2991819277}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2991819277}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:38.079 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14604:up:rejoin 2021-11-20T06:06:38.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:38 smithi139 conmon[33501]: audit 2021-11-20T06:06:38.079565+0000 mon.smithi139 (mon.0) 731 : audit [DBG] 2021-11-20T06:06:38.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:38 smithi139 conmon[33501]: from='client.? 172.21.15.139:0/1261295227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:38.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:38 smithi178 conmon[36600]: audit 2021-11-20T06:06:38.079565+0000 mon.smithi139 (mon.0) 731 : audit [DBG] from='client.? 172.21.15.139:0/1261295227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:39.080 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:39.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.222+0000 7fe62c00e700 1 -- 172.21.15.139:0/1057416425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6240f7940 msgr2=0x7fe6240f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:39.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.222+0000 7fe62c00e700 1 --2- 172.21.15.139:0/1057416425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6240f7940 0x7fe6240f7da0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fe618004660 tx=0x7fe618009b30).stop 2021-11-20T06:06:39.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.222+0000 7fe62c00e700 1 -- 172.21.15.139:0/1057416425 shutdown_connections 2021-11-20T06:06:39.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.222+0000 7fe62c00e700 1 --2- 172.21.15.139:0/1057416425 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6240f7940 0x7fe6240f7da0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:39.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.222+0000 7fe62c00e700 1 --2- 172.21.15.139:0/1057416425 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6240f66a0 0x7fe6240f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:39.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.222+0000 7fe62c00e700 1 -- 172.21.15.139:0/1057416425 >> 172.21.15.139:0/1057416425 conn(0x7fe6240f1da0 msgr2=0x7fe6240f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:39.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.222+0000 7fe62c00e700 1 -- 172.21.15.139:0/1057416425 shutdown_connections 2021-11-20T06:06:39.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.222+0000 7fe62c00e700 1 -- 172.21.15.139:0/1057416425 wait complete. 2021-11-20T06:06:39.211 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.222+0000 7fe62c00e700 1 Processor -- start 2021-11-20T06:06:39.211 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.222+0000 7fe62c00e700 1 -- start start 2021-11-20T06:06:39.211 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe62c00e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6240f66a0 0x7fe624106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:39.211 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe62c00e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6240f7940 0x7fe6241073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:39.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe62c00e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe624107960 con 0x7fe6240f66a0 2021-11-20T06:06:39.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe62c00e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe624107aa0 con 0x7fe6240f7940 2021-11-20T06:06:39.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe629daa700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6240f66a0 0x7fe624106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:39.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe629daa700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6240f66a0 0x7fe624106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58370/0 (socket says 172.21.15.139:58370) 2021-11-20T06:06:39.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe629daa700 1 -- 172.21.15.139:0/370151794 learned_addr learned my addr 172.21.15.139:0/370151794 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:39.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe6295a9700 1 --2- 172.21.15.139:0/370151794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6240f7940 0x7fe6241073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:39.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe629daa700 1 -- 172.21.15.139:0/370151794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6240f7940 msgr2=0x7fe6241073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:39.214 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe629daa700 1 --2- 172.21.15.139:0/370151794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6240f7940 0x7fe6241073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:39.214 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe629daa700 1 -- 172.21.15.139:0/370151794 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe618005040 con 0x7fe6240f66a0 2021-11-20T06:06:39.215 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe629daa700 1 --2- 172.21.15.139:0/370151794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6240f66a0 0x7fe624106ec0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fe61400c710 tx=0x7fe614004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:39.215 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe6227fc700 1 -- 172.21.15.139:0/370151794 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6140106f0 con 0x7fe6240f66a0 2021-11-20T06:06:39.215 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe62c00e700 1 -- 172.21.15.139:0/370151794 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe62410c540 con 0x7fe6240f66a0 2021-11-20T06:06:39.215 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe6227fc700 1 -- 172.21.15.139:0/370151794 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe614010d00 con 0x7fe6240f66a0 2021-11-20T06:06:39.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe62c00e700 1 -- 172.21.15.139:0/370151794 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe62410cb00 con 0x7fe6240f66a0 2021-11-20T06:06:39.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.223+0000 7fe6227fc700 1 -- 172.21.15.139:0/370151794 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe614012d90 con 0x7fe6240f66a0 2021-11-20T06:06:39.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.224+0000 7fe62c00e700 1 -- 172.21.15.139:0/370151794 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe624027a00 con 0x7fe6240f66a0 2021-11-20T06:06:39.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.225+0000 7fe6227fc700 1 -- 172.21.15.139:0/370151794 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe614010070 con 0x7fe6240f66a0 2021-11-20T06:06:39.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.225+0000 7fe6227fc700 1 --2- 172.21.15.139:0/370151794 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe60c05c2a0 0x7fe60c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:39.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.225+0000 7fe6227fc700 1 -- 172.21.15.139:0/370151794 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fe6140767b0 con 0x7fe6240f66a0 2021-11-20T06:06:39.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.225+0000 7fe6295a9700 1 --2- 172.21.15.139:0/370151794 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe60c05c2a0 0x7fe60c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:39.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.225+0000 7fe6295a9700 1 --2- 172.21.15.139:0/370151794 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe60c05c2a0 0x7fe60c05e760 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fe618013580 tx=0x7fe618009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:39.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.229+0000 7fe6227fc700 1 -- 172.21.15.139:0/370151794 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe614049110 con 0x7fe6240f66a0 2021-11-20T06:06:39.399 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.414+0000 7fe62c00e700 1 -- 172.21.15.139:0/370151794 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6240fee30 con 0x7fe6240f66a0 2021-11-20T06:06:39.399 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.415+0000 7fe6227fc700 1 -- 172.21.15.139:0/370151794 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2940 (secure 0 0 0) 0x7fe61404c730 con 0x7fe6240f66a0 2021-11-20T06:06:39.400 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:39.400 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:24.227400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14604":{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:39.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.416+0000 7fe62c00e700 1 -- 172.21.15.139:0/370151794 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe60c05c2a0 msgr2=0x7fe60c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:39.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.416+0000 7fe62c00e700 1 --2- 172.21.15.139:0/370151794 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe60c05c2a0 0x7fe60c05e760 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fe618013580 tx=0x7fe618009b30).stop 2021-11-20T06:06:39.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.416+0000 7fe62c00e700 1 -- 172.21.15.139:0/370151794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6240f66a0 msgr2=0x7fe624106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:39.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.416+0000 7fe62c00e700 1 --2- 172.21.15.139:0/370151794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6240f66a0 0x7fe624106ec0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fe61400c710 tx=0x7fe614004d10).stop 2021-11-20T06:06:39.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.416+0000 7fe62c00e700 1 -- 172.21.15.139:0/370151794 shutdown_connections 2021-11-20T06:06:39.403 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.416+0000 7fe62c00e700 1 --2- 172.21.15.139:0/370151794 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe60c05c2a0 0x7fe60c05e760 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:39.403 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.416+0000 7fe62c00e700 1 --2- 172.21.15.139:0/370151794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe6240f66a0 0x7fe624106ec0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:39.403 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.416+0000 7fe62c00e700 1 --2- 172.21.15.139:0/370151794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6240f7940 0x7fe6241073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:39.403 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.416+0000 7fe62c00e700 1 -- 172.21.15.139:0/370151794 >> 172.21.15.139:0/370151794 conn(0x7fe6240f1da0 msgr2=0x7fe6240fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:39.403 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.416+0000 7fe62c00e700 1 -- 172.21.15.139:0/370151794 shutdown_connections 2021-11-20T06:06:39.404 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:39.416+0000 7fe62c00e700 1 -- 172.21.15.139:0/370151794 wait complete. 2021-11-20T06:06:39.404 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 61 2021-11-20T06:06:39.411 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:39 smithi139 conmon[33501]: cluster 2021-11-20T06:06:38.259893+0000 mgr.smithi139.dplyxn (mgr.14162) 285 : cluster [DBG] pgmap v256: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 341 B/s rd, 0 op/s 2021-11-20T06:06:39.413 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:24.227400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14604}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14604': {'gid': 14604, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 58, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6829/2991819277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2991819277}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2991819277}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:39.413 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14604:up:rejoin 2021-11-20T06:06:39.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:39 smithi178 conmon[36600]: cluster 2021-11-20T06:06:38.259893+0000 mgr.smithi139.dplyxn (mgr.14162) 285 : cluster [DBG] pgmap v256: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 341 B/s rd, 0 op/s 2021-11-20T06:06:40.414 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:40.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.556+0000 7fd2f54e8700 1 -- 172.21.15.139:0/2656782398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f0060730 msgr2=0x7fd2f00f92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:40.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.556+0000 7fd2f54e8700 1 --2- 172.21.15.139:0/2656782398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f0060730 0x7fd2f00f92a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fd2e0004660 tx=0x7fd2e0009b30).stop 2021-11-20T06:06:40.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.556+0000 7fd2f54e8700 1 -- 172.21.15.139:0/2656782398 shutdown_connections 2021-11-20T06:06:40.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.556+0000 7fd2f54e8700 1 --2- 172.21.15.139:0/2656782398 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2f00f9820 0x7fd2f00fbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:40.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.556+0000 7fd2f54e8700 1 --2- 172.21.15.139:0/2656782398 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f0060730 0x7fd2f00f92a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:40.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.556+0000 7fd2f54e8700 1 -- 172.21.15.139:0/2656782398 >> 172.21.15.139:0/2656782398 conn(0x7fd2f00f0ce0 msgr2=0x7fd2f00f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:40.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.556+0000 7fd2f54e8700 1 -- 172.21.15.139:0/2656782398 shutdown_connections 2021-11-20T06:06:40.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.556+0000 7fd2f54e8700 1 -- 172.21.15.139:0/2656782398 wait complete. 2021-11-20T06:06:40.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.556+0000 7fd2f54e8700 1 Processor -- start 2021-11-20T06:06:40.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.556+0000 7fd2f54e8700 1 -- start start 2021-11-20T06:06:40.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2f54e8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2f0060730 0x7fd2f0106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:40.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2f54e8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f00f9820 0x7fd2f01073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:40.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2f54e8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f0107990 con 0x7fd2f00f9820 2021-11-20T06:06:40.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2f54e8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f0107ad0 con 0x7fd2f0060730 2021-11-20T06:06:40.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2ee7fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f00f9820 0x7fd2f01073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:40.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2ee7fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f00f9820 0x7fd2f01073e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58382/0 (socket says 172.21.15.139:58382) 2021-11-20T06:06:40.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2ee7fc700 1 -- 172.21.15.139:0/2424502457 learned_addr learned my addr 172.21.15.139:0/2424502457 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:40.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2eeffd700 1 --2- 172.21.15.139:0/2424502457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2f0060730 0x7fd2f0106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:40.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2ee7fc700 1 -- 172.21.15.139:0/2424502457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2f0060730 msgr2=0x7fd2f0106ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:40.548 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2ee7fc700 1 --2- 172.21.15.139:0/2424502457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2f0060730 0x7fd2f0106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:40.548 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2ee7fc700 1 -- 172.21.15.139:0/2424502457 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e0005040 con 0x7fd2f00f9820 2021-11-20T06:06:40.548 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2ee7fc700 1 --2- 172.21.15.139:0/2424502457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f00f9820 0x7fd2f01073e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fd2e400c510 tx=0x7fd2e4007df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:40.548 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2d77fe700 1 -- 172.21.15.139:0/2424502457 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2e400f650 con 0x7fd2f00f9820 2021-11-20T06:06:40.549 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2d77fe700 1 -- 172.21.15.139:0/2424502457 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2e400fc60 con 0x7fd2f00f9820 2021-11-20T06:06:40.549 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2d77fe700 1 -- 172.21.15.139:0/2424502457 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2e4012cb0 con 0x7fd2f00f9820 2021-11-20T06:06:40.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2f54e8700 1 -- 172.21.15.139:0/2424502457 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f010c570 con 0x7fd2f00f9820 2021-11-20T06:06:40.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.557+0000 7fd2f54e8700 1 -- 172.21.15.139:0/2424502457 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f010cb30 con 0x7fd2f00f9820 2021-11-20T06:06:40.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.558+0000 7fd2f54e8700 1 -- 172.21.15.139:0/2424502457 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f0027a00 con 0x7fd2f00f9820 2021-11-20T06:06:40.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.559+0000 7fd2d77fe700 1 -- 172.21.15.139:0/2424502457 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd2e4010070 con 0x7fd2f00f9820 2021-11-20T06:06:40.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.559+0000 7fd2d77fe700 1 --2- 172.21.15.139:0/2424502457 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd2d805c2a0 0x7fd2d805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:40.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.559+0000 7fd2d77fe700 1 -- 172.21.15.139:0/2424502457 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fd2e4075f50 con 0x7fd2f00f9820 2021-11-20T06:06:40.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.559+0000 7fd2eeffd700 1 --2- 172.21.15.139:0/2424502457 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd2d805c2a0 0x7fd2d805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:40.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.559+0000 7fd2eeffd700 1 --2- 172.21.15.139:0/2424502457 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd2d805c2a0 0x7fd2d805e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fd2e0009d60 tx=0x7fd2e0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:40.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.563+0000 7fd2d77fe700 1 -- 172.21.15.139:0/2424502457 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2e40452b0 con 0x7fd2f00f9820 2021-11-20T06:06:40.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:40 smithi178 conmon[36600]: audit 2021-11-20T06:06:39.415522+0000 mon.smithi139 (mon.0) 732 : audit [DBG] from='client.? 172.21.15.139:0/370151794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:40.730 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.746+0000 7fd2f54e8700 1 -- 172.21.15.139:0/2424502457 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2f00fee30 con 0x7fd2f00f9820 2021-11-20T06:06:40.731 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.746+0000 7fd2d77fe700 1 -- 172.21.15.139:0/2424502457 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2940 (secure 0 0 0) 0x7fd2e404c2e0 con 0x7fd2f00f9820 2021-11-20T06:06:40.732 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:40.732 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:24.227400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14604":{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:40.733 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.748+0000 7fd2f54e8700 1 -- 172.21.15.139:0/2424502457 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd2d805c2a0 msgr2=0x7fd2d805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:40.734 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.748+0000 7fd2f54e8700 1 --2- 172.21.15.139:0/2424502457 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd2d805c2a0 0x7fd2d805e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fd2e0009d60 tx=0x7fd2e0009b30).stop 2021-11-20T06:06:40.734 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.748+0000 7fd2f54e8700 1 -- 172.21.15.139:0/2424502457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f00f9820 msgr2=0x7fd2f01073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:40.734 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.748+0000 7fd2f54e8700 1 --2- 172.21.15.139:0/2424502457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f00f9820 0x7fd2f01073e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fd2e400c510 tx=0x7fd2e4007df0).stop 2021-11-20T06:06:40.734 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.748+0000 7fd2f54e8700 1 -- 172.21.15.139:0/2424502457 shutdown_connections 2021-11-20T06:06:40.735 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.748+0000 7fd2f54e8700 1 --2- 172.21.15.139:0/2424502457 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd2d805c2a0 0x7fd2d805e760 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:40.735 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.748+0000 7fd2f54e8700 1 --2- 172.21.15.139:0/2424502457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd2f0060730 0x7fd2f0106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:40.735 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.748+0000 7fd2f54e8700 1 --2- 172.21.15.139:0/2424502457 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2f00f9820 0x7fd2f01073e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:40.736 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.748+0000 7fd2f54e8700 1 -- 172.21.15.139:0/2424502457 >> 172.21.15.139:0/2424502457 conn(0x7fd2f00f0ce0 msgr2=0x7fd2f00f30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:40.736 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.748+0000 7fd2f54e8700 1 -- 172.21.15.139:0/2424502457 shutdown_connections 2021-11-20T06:06:40.736 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:40.748+0000 7fd2f54e8700 1 -- 172.21.15.139:0/2424502457 wait complete. 2021-11-20T06:06:40.736 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 61 2021-11-20T06:06:40.743 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:40 smithi139 conmon[33501]: audit 2021-11-20T06:06:39.415522+0000 mon.smithi139 (mon.0) 732 : audit [DBG] from='client.? 172.21.15.139:0/370151794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:40.744 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:40 smithi139 conmon[33501]: 2021-11-20T06:06:40.745 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:24.227400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14604}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14604': {'gid': 14604, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 58, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6829/2991819277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2991819277}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2991819277}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:40.746 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14604:up:rejoin 2021-11-20T06:06:41.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:41 smithi178 conmon[36600]: cluster 2021-11-20T06:06:40.260557+0000 mgr.smithi139.dplyxn (mgr.14162) 286 : cluster [DBG] pgmap v257: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:06:41.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:41 smithi178 conmon[36600]: audit 2021-11-20T06:06:40.747131+0000 mon.smithi139 (mon.0) 733 : audit [DBG] from='client.? 172.21.15.139:0/2424502457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:41.747 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:41.772 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:41 smithi139 conmon[33501]: cluster 2021-11-20T06:06:40.260557+0000 mgr.smithi139.dplyxn (mgr.14162) 286 : cluster 2021-11-20T06:06:41.772 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:41 smithi139 conmon[33501]: [DBG] pgmap v257: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:06:41.773 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:41 smithi139 conmon[33501]: audit 2021-11-20T06:06:40.747131+0000 mon.smithi139 (mon.0) 733 : audit [DBG] from='client.? 172.21.15.139:0/2424502457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:41.869 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.882+0000 7fe7d6cf0700 1 -- 172.21.15.139:0/925883189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7d00f54c0 msgr2=0x7fe7d00f58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:41.869 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.882+0000 7fe7d6cf0700 1 --2- 172.21.15.139:0/925883189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7d00f54c0 0x7fe7d00f58e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fe7cc004660 tx=0x7fe7cc009b30).stop 2021-11-20T06:06:41.870 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.882+0000 7fe7d6cf0700 1 -- 172.21.15.139:0/925883189 shutdown_connections 2021-11-20T06:06:41.870 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.882+0000 7fe7d6cf0700 1 --2- 172.21.15.139:0/925883189 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7d00f6760 0x7fe7d00f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:41.870 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.882+0000 7fe7d6cf0700 1 --2- 172.21.15.139:0/925883189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7d00f54c0 0x7fe7d00f58e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:41.870 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.882+0000 7fe7d6cf0700 1 -- 172.21.15.139:0/925883189 >> 172.21.15.139:0/925883189 conn(0x7fe7d00f0c00 msgr2=0x7fe7d00f3040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:41.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.882+0000 7fe7d6cf0700 1 -- 172.21.15.139:0/925883189 shutdown_connections 2021-11-20T06:06:41.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.882+0000 7fe7d6cf0700 1 -- 172.21.15.139:0/925883189 wait complete. 2021-11-20T06:06:41.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d6cf0700 1 Processor -- start 2021-11-20T06:06:41.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d6cf0700 1 -- start start 2021-11-20T06:06:41.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d6cf0700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7d00f54c0 0x7fe7d0105d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:41.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d6cf0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7d00f6760 0x7fe7d0106220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:41.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d6cf0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d01067d0 con 0x7fe7d00f6760 2021-11-20T06:06:41.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d6cf0700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d0106fd0 con 0x7fe7d00f54c0 2021-11-20T06:06:41.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d54ed700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7d00f6760 0x7fe7d0106220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:41.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d54ed700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7d00f6760 0x7fe7d0106220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58394/0 (socket says 172.21.15.139:58394) 2021-11-20T06:06:41.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d54ed700 1 -- 172.21.15.139:0/3877663793 learned_addr learned my addr 172.21.15.139:0/3877663793 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:41.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d54ed700 1 -- 172.21.15.139:0/3877663793 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7d00f54c0 msgr2=0x7fe7d0105d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:41.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d5cee700 1 --2- 172.21.15.139:0/3877663793 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7d00f54c0 0x7fe7d0105d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:41.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d54ed700 1 --2- 172.21.15.139:0/3877663793 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7d00f54c0 0x7fe7d0105d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:41.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d54ed700 1 -- 172.21.15.139:0/3877663793 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7cc005040 con 0x7fe7d00f6760 2021-11-20T06:06:41.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d54ed700 1 --2- 172.21.15.139:0/3877663793 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7d00f6760 0x7fe7d0106220 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fe7c00026d0 tx=0x7fe7c0009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:41.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7c67fc700 1 -- 172.21.15.139:0/3877663793 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7c0009320 con 0x7fe7d00f6760 2021-11-20T06:06:41.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7c67fc700 1 -- 172.21.15.139:0/3877663793 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe7c000f040 con 0x7fe7d00f6760 2021-11-20T06:06:41.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d6cf0700 1 -- 172.21.15.139:0/3877663793 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d01072b0 con 0x7fe7d00f6760 2021-11-20T06:06:41.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7c67fc700 1 -- 172.21.15.139:0/3877663793 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7c00147e0 con 0x7fe7d00f6760 2021-11-20T06:06:41.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.883+0000 7fe7d6cf0700 1 -- 172.21.15.139:0/3877663793 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d0107870 con 0x7fe7d00f6760 2021-11-20T06:06:41.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.884+0000 7fe7d6cf0700 1 -- 172.21.15.139:0/3877663793 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d00f48b0 con 0x7fe7d00f6760 2021-11-20T06:06:41.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.885+0000 7fe7c67fc700 1 -- 172.21.15.139:0/3877663793 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe7c0009480 con 0x7fe7d00f6760 2021-11-20T06:06:41.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.885+0000 7fe7c67fc700 1 --2- 172.21.15.139:0/3877663793 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe7bc05c300 0x7fe7bc05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:41.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.885+0000 7fe7c67fc700 1 -- 172.21.15.139:0/3877663793 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fe7c0076910 con 0x7fe7d00f6760 2021-11-20T06:06:41.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.885+0000 7fe7d5cee700 1 --2- 172.21.15.139:0/3877663793 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe7bc05c300 0x7fe7bc05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:41.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.886+0000 7fe7d5cee700 1 --2- 172.21.15.139:0/3877663793 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe7bc05c300 0x7fe7bc05e7c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fe7cc009d60 tx=0x7fe7cc009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:41.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:41.889+0000 7fe7c67fc700 1 -- 172.21.15.139:0/3877663793 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7c0049270 con 0x7fe7d00f6760 2021-11-20T06:06:42.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:42.076+0000 7fe7d6cf0700 1 -- 172.21.15.139:0/3877663793 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7d0045ad0 con 0x7fe7d00f6760 2021-11-20T06:06:42.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:42.077+0000 7fe7c67fc700 1 -- 172.21.15.139:0/3877663793 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2940 (secure 0 0 0) 0x7fe7c004c890 con 0x7fe7d00f6760 2021-11-20T06:06:42.063 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:42.063 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61},{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:24.227400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14604":{"gid":14604,"name":"cephfs.smithi139.oudezh","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6829/2991819277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2991819277},{"type":"v1","addr":"172.21.15.139:6829","nonce":2991819277}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:42.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:42.079+0000 7fe7d6cf0700 1 -- 172.21.15.139:0/3877663793 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe7bc05c300 msgr2=0x7fe7bc05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:42.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:42.079+0000 7fe7d6cf0700 1 --2- 172.21.15.139:0/3877663793 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe7bc05c300 0x7fe7bc05e7c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fe7cc009d60 tx=0x7fe7cc009b30).stop 2021-11-20T06:06:42.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:42.079+0000 7fe7d6cf0700 1 -- 172.21.15.139:0/3877663793 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7d00f6760 msgr2=0x7fe7d0106220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:42.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:42.079+0000 7fe7d6cf0700 1 --2- 172.21.15.139:0/3877663793 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7d00f6760 0x7fe7d0106220 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fe7c00026d0 tx=0x7fe7c0009de0).stop 2021-11-20T06:06:42.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:42.079+0000 7fe7d6cf0700 1 -- 172.21.15.139:0/3877663793 shutdown_connections 2021-11-20T06:06:42.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:42.079+0000 7fe7d6cf0700 1 --2- 172.21.15.139:0/3877663793 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fe7bc05c300 0x7fe7bc05e7c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:42.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:42.079+0000 7fe7d6cf0700 1 --2- 172.21.15.139:0/3877663793 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7d00f54c0 0x7fe7d0105d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:42.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:42.079+0000 7fe7d6cf0700 1 --2- 172.21.15.139:0/3877663793 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7d00f6760 0x7fe7d0106220 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:42.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:42.079+0000 7fe7d6cf0700 1 -- 172.21.15.139:0/3877663793 >> 172.21.15.139:0/3877663793 conn(0x7fe7d00f0c00 msgr2=0x7fe7d00fa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:42.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:42.079+0000 7fe7d6cf0700 1 -- 172.21.15.139:0/3877663793 shutdown_connections 2021-11-20T06:06:42.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:42.079+0000 7fe7d6cf0700 1 -- 172.21.15.139:0/3877663793 wait complete. 2021-11-20T06:06:42.066 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 61 2021-11-20T06:06:42.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:24.227400+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14604}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14604': {'gid': 14604, 'name': 'cephfs.smithi139.oudezh', 'rank': 0, 'incarnation': 58, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6829/2991819277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6828', 'nonce': 2991819277}, {'type': 'v1', 'addr': '172.21.15.139:6829', 'nonce': 2991819277}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:42.076 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14604:up:rejoin 2021-11-20T06:06:42.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:42 smithi178 conmon[36600]: audit 2021-11-20T06:06:42.077802+0000 mon.smithi139 (mon.0) 734 : audit [DBG] from='client.? 172.21.15.139:0/3877663793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:42.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:42 smithi178 conmon[36600]: cluster 2021-11-20T06:06:42.220091+0000 mon.smithi139 (mon.0) 735 : cluster [WRN] Replacing daemon mds.cephfs.smithi139.oudezh as rank 0 with standby daemon mds.cephfs.smithi139.riczqb 2021-11-20T06:06:42.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:42 smithi178 conmon[36600]: cluster 2021-11-20T06:06:42.220145+0000 mon.smithi139 (mon.0) 736 : cluster [INF] MDS daemon mds.cephfs.smithi139.oudezh is removed because it is dead or otherwise unavailable. 2021-11-20T06:06:42.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:42 smithi178 conmon[36600]: cluster 2021-11-20T06:06: 2021-11-20T06:06:42.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:42 smithi178 conmon[36600]: 42.225889+0000 mon.smithi139 (mon.0) 737 : cluster [DBG] osdmap e52: 6 total, 6 up, 6 in 2021-11-20T06:06:42.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:42 smithi178 conmon[36600]: cluster 2021-11-20T06:06:42.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:42 smithi178 conmon[36600]: 2021-11-20T06:06:42.231045+0000 mon.smithi139 (mon 2021-11-20T06:06:42.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:42 smithi178 conmon[36600]: .0) 738 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:replay} 2 up:standby 2021-11-20T06:06:42.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:42 smithi139 conmon[33501]: audit 2021-11-20T06:06:42.077802+0000 mon.smithi139 (mon.0) 734 : 2021-11-20T06:06:42.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:42 smithi139 conmon[33501]: audit [DBG] from='client.? 172.21.15.139:0/3877663793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:42.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:42 smithi139 conmon[33501]: cluster 2021-11-20T06:06:42.220091+0000 mon.smithi139 (mon.0) 735 : cluster [WRN] Replacing daemon mds.cephfs.smithi139.oudezh as rank 0 with standby daemon mds.cephfs.smithi139.riczqb 2021-11-20T06:06:42.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:42 smithi139 conmon[33501]: cluster 2021-11-20T 2021-11-20T06:06:42.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:42 smithi139 conmon[33501]: 06:06:42.220145+0000 mon.smithi139 (mon.0) 736 : cluster [INF] MDS daemon mds.cephfs.smithi139.oudezh is removed because it is dead or otherwise unavailable. 2021-11-20T06:06:42.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:42 smithi139 conmon[33501]: cluster 2021-11-20T06:06:42.225889+0000 mon.smithi139 2021-11-20T06:06:42.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:42 smithi139 conmon[33501]: (mon.0) 737 : cluster [DBG] osdmap e52: 6 total, 6 up, 6 in 2021-11-20T06:06:42.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:42 smithi139 conmon[33501]: cluster 2021-11-20T06:06:42.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:42 smithi139 conmon[33501]: 2021-11-20T06:06: 2021-11-20T06:06:42.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:42 smithi139 conmon[33501]: 42.231045+0000 mon.smithi139 (mon. 2021-11-20T06:06:42.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:42 smithi139 conmon[33501]: 0) 738 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:replay} 2 up:standby 2021-11-20T06:06:43.077 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:43.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.212+0000 7f69296d7700 1 -- 172.21.15.139:0/1067829969 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69240f6410 msgr2=0x7f69240f6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:43.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.212+0000 7f69296d7700 1 --2- 172.21.15.139:0/1067829969 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69240f6410 0x7f69240f6870 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f6918004660 tx=0x7f6918009b30).stop 2021-11-20T06:06:43.199 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.212+0000 7f69296d7700 1 -- 172.21.15.139:0/1067829969 shutdown_connections 2021-11-20T06:06:43.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.212+0000 7f69296d7700 1 --2- 172.21.15.139:0/1067829969 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69240f6410 0x7f69240f6870 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:43.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.212+0000 7f69296d7700 1 --2- 172.21.15.139:0/1067829969 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69240f5a70 0x7f69240f5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:43.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.212+0000 7f69296d7700 1 -- 172.21.15.139:0/1067829969 >> 172.21.15.139:0/1067829969 conn(0x7f69240f0360 msgr2=0x7f69240f27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:43.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.212+0000 7f69296d7700 1 -- 172.21.15.139:0/1067829969 shutdown_connections 2021-11-20T06:06:43.200 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.212+0000 7f69296d7700 1 -- 172.21.15.139:0/1067829969 wait complete. 2021-11-20T06:06:43.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.213+0000 7f69296d7700 1 Processor -- start 2021-11-20T06:06:43.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.213+0000 7f69296d7700 1 -- start start 2021-11-20T06:06:43.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.213+0000 7f69296d7700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69240f5a70 0x7f6924106d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:43.201 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.213+0000 7f69296d7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69240f6410 0x7f6924107270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:43.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.213+0000 7f69296d7700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6924107870 con 0x7f69240f6410 2021-11-20T06:06:43.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.213+0000 7f69296d7700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69241080b0 con 0x7f69240f5a70 2021-11-20T06:06:43.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.213+0000 7f69237fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69240f6410 0x7f6924107270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:43.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.213+0000 7f69237fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69240f6410 0x7f6924107270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58414/0 (socket says 172.21.15.139:58414) 2021-11-20T06:06:43.202 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.213+0000 7f69237fe700 1 -- 172.21.15.139:0/2715593366 learned_addr learned my addr 172.21.15.139:0/2715593366 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:43.203 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.213+0000 7f6923fff700 1 --2- 172.21.15.139:0/2715593366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69240f5a70 0x7f6924106d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:43.203 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.214+0000 7f69237fe700 1 -- 172.21.15.139:0/2715593366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69240f5a70 msgr2=0x7f6924106d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:43.203 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.214+0000 7f69237fe700 1 --2- 172.21.15.139:0/2715593366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69240f5a70 0x7f6924106d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:43.203 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.214+0000 7f69237fe700 1 -- 172.21.15.139:0/2715593366 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6918005040 con 0x7f69240f6410 2021-11-20T06:06:43.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.214+0000 7f69237fe700 1 --2- 172.21.15.139:0/2715593366 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69240f6410 0x7f6924107270 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f6918009830 tx=0x7f6918009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:43.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.214+0000 7f6920ff9700 1 -- 172.21.15.139:0/2715593366 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6918018070 con 0x7f69240f6410 2021-11-20T06:06:43.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.214+0000 7f6920ff9700 1 -- 172.21.15.139:0/2715593366 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6918013410 con 0x7f69240f6410 2021-11-20T06:06:43.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.214+0000 7f69296d7700 1 -- 172.21.15.139:0/2715593366 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6924108330 con 0x7f69240f6410 2021-11-20T06:06:43.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.214+0000 7f6920ff9700 1 -- 172.21.15.139:0/2715593366 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f691801c400 con 0x7f69240f6410 2021-11-20T06:06:43.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.214+0000 7f69296d7700 1 -- 172.21.15.139:0/2715593366 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6924108810 con 0x7f69240f6410 2021-11-20T06:06:43.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.215+0000 7f69296d7700 1 -- 172.21.15.139:0/2715593366 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69240f4010 con 0x7f69240f6410 2021-11-20T06:06:43.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.215+0000 7f6920ff9700 1 -- 172.21.15.139:0/2715593366 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f691801d420 con 0x7f69240f6410 2021-11-20T06:06:43.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.215+0000 7f6920ff9700 1 --2- 172.21.15.139:0/2715593366 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f690c05c300 0x7f690c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:43.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.216+0000 7f6920ff9700 1 -- 172.21.15.139:0/2715593366 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f69180781e0 con 0x7f69240f6410 2021-11-20T06:06:43.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.216+0000 7f6923fff700 1 --2- 172.21.15.139:0/2715593366 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f690c05c300 0x7f690c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:43.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.216+0000 7f6923fff700 1 --2- 172.21.15.139:0/2715593366 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f690c05c300 0x7f690c05e7c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f6924107b40 tx=0x7f6914009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:43.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.219+0000 7f6920ff9700 1 -- 172.21.15.139:0/2715593366 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f691804aa30 con 0x7f69240f6410 2021-11-20T06:06:43.401 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.416+0000 7f69296d7700 1 -- 172.21.15.139:0/2715593366 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6924108f10 con 0x7f69240f6410 2021-11-20T06:06:43.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.417+0000 7f6920ff9700 1 -- 172.21.15.139:0/2715593366 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2565 (secure 0 0 0) 0x7f691804e050 con 0x7f69240f6410 2021-11-20T06:06:43.402 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:43.402 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":63,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:43.229591+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14660":{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":62,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:43.404 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.418+0000 7f69296d7700 1 -- 172.21.15.139:0/2715593366 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f690c05c300 msgr2=0x7f690c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:43.404 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.418+0000 7f69296d7700 1 --2- 172.21.15.139:0/2715593366 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f690c05c300 0x7f690c05e7c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f6924107b40 tx=0x7f6914009040).stop 2021-11-20T06:06:43.404 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.418+0000 7f69296d7700 1 -- 172.21.15.139:0/2715593366 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69240f6410 msgr2=0x7f6924107270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:43.404 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.419+0000 7f69296d7700 1 --2- 172.21.15.139:0/2715593366 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69240f6410 0x7f6924107270 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f6918009830 tx=0x7f6918009b30).stop 2021-11-20T06:06:43.405 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.419+0000 7f69296d7700 1 -- 172.21.15.139:0/2715593366 shutdown_connections 2021-11-20T06:06:43.405 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.419+0000 7f69296d7700 1 --2- 172.21.15.139:0/2715593366 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f690c05c300 0x7f690c05e7c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:43.405 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.419+0000 7f69296d7700 1 --2- 172.21.15.139:0/2715593366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69240f5a70 0x7f6924106d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:43.405 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.419+0000 7f69296d7700 1 --2- 172.21.15.139:0/2715593366 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f69240f6410 0x7f6924107270 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:43.405 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.419+0000 7f69296d7700 1 -- 172.21.15.139:0/2715593366 >> 172.21.15.139:0/2715593366 conn(0x7f69240f0360 msgr2=0x7f69240f4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:43.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.419+0000 7f69296d7700 1 -- 172.21.15.139:0/2715593366 shutdown_connections 2021-11-20T06:06:43.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:43.419+0000 7f69296d7700 1 -- 172.21.15.139:0/2715593366 wait complete. 2021-11-20T06:06:43.406 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 63 2021-11-20T06:06:43.414 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:43 smithi139 conmon[33501]: cluster 2021-11-20T06:06:42.260889+0000 mgr.smithi139.dplyxn (mgr.14162) 287 : 2021-11-20T06:06:43.414 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:43 smithi139 conmon[33501]: cluster [DBG] pgmap v259: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:06:43.414 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:43 smithi139 conmon[33501]: cluster 2021-11-20T06:06:43.234965+0000 mon.smithi139 (mon.0) 739 : cluster [DBG] mds.? [v2:172.21.15.139:6826/811251312,v1:172.21.15.139:6827/811251312] up:reconnect 2021-11-20T06:06:43.414 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:43 smithi139 conmon[33501]: cluster 2021-11-20T06:06:43.235025+0000 mon.smithi139 (mon.0) 740 2021-11-20T06:06:43.415 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:43 smithi139 conmon[33501]: : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:reconnect} 2 up:standby 2021-11-20T06:06:43.416 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:43.229591+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14660': {'gid': 14660, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 62, 'state': 'up:reconnect', 'state_seq': 7, 'addr': '172.21.15.139:6827/811251312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 811251312}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 811251312}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:43.416 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14660:up:reconnect 2021-11-20T06:06:43.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:43 smithi178 conmon[36600]: cluster 2021-11-20T06:06:42.260889+0000 mgr.smithi139.dplyxn (mgr.14162) 287 : cluster [DBG] pgmap v259: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:06:43.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:43 smithi178 conmon[36600]: cluster 2021-11-20T06:06: 2021-11-20T06:06:43.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:43 smithi178 conmon[36600]: 43.234965+0000 mon.smithi139 (mon.0) 739 : cluster [DBG] mds.? [v2:172.21.15.139:6826/811251312,v1:172.21.15.139:6827/811251312] up:reconnect 2021-11-20T06:06:43.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:43 smithi178 conmon[36600]: cluster 2021-11-20T06:06:43.235025+0000 mon.smithi139 (mon.0) 740 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:reconnect} 2 up:standby 2021-11-20T06:06:44.417 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:44.444 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:44 smithi139 conmon[33501]: audit 2021-11-20T06:06:43.417728+0000 mon.smithi139 (mon.0) 741 : 2021-11-20T06:06:44.445 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:44 smithi139 conmon[33501]: audit [DBG] from='client.? 172.21.15.139:0/2715593366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:44.445 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:44 smithi139 conmon[33501]: cluster 2021-11-20T06:06:44.236901+0000 mon.smithi139 (mon.0) 742 : cluster [DBG] mds.? [v2:172.21.15.139:6826/811251312,v1:172.21.15.139:6827/811251312] up:rejoin 2021-11-20T06:06:44.445 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:44 smithi139 conmon[33501]: cluster 2021-11-20T06:06:44.236969+0000 mon.smithi139 (mon.0) 743 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 2 up:standby 2021-11-20T06:06:44.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.553+0000 7fa350be5700 1 -- 172.21.15.139:0/459124875 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c0f6670 msgr2=0x7fa34c0f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:44.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.553+0000 7fa350be5700 1 --2- 172.21.15.139:0/459124875 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c0f6670 0x7fa34c0f6a90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fa33c004660 tx=0x7fa33c009b30).stop 2021-11-20T06:06:44.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.553+0000 7fa350be5700 1 -- 172.21.15.139:0/459124875 shutdown_connections 2021-11-20T06:06:44.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.553+0000 7fa350be5700 1 --2- 172.21.15.139:0/459124875 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa34c0f7910 0x7fa34c0f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:44.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.553+0000 7fa350be5700 1 --2- 172.21.15.139:0/459124875 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c0f6670 0x7fa34c0f6a90 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:44.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.553+0000 7fa350be5700 1 -- 172.21.15.139:0/459124875 >> 172.21.15.139:0/459124875 conn(0x7fa34c0f1db0 msgr2=0x7fa34c0f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:44.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.553+0000 7fa350be5700 1 -- 172.21.15.139:0/459124875 shutdown_connections 2021-11-20T06:06:44.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.553+0000 7fa350be5700 1 -- 172.21.15.139:0/459124875 wait complete. 2021-11-20T06:06:44.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.553+0000 7fa350be5700 1 Processor -- start 2021-11-20T06:06:44.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.553+0000 7fa350be5700 1 -- start start 2021-11-20T06:06:44.548 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.553+0000 7fa350be5700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa34c0f6670 0x7fa34c106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:44.548 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.553+0000 7fa350be5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c0f7910 0x7fa34c1073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:44.549 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.553+0000 7fa350be5700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c107960 con 0x7fa34c0f7910 2021-11-20T06:06:44.549 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.553+0000 7fa350be5700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c107aa0 con 0x7fa34c0f6670 2021-11-20T06:06:44.549 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.554+0000 7fa349d9b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c0f7910 0x7fa34c1073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:44.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.554+0000 7fa349d9b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c0f7910 0x7fa34c1073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58426/0 (socket says 172.21.15.139:58426) 2021-11-20T06:06:44.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.554+0000 7fa349d9b700 1 -- 172.21.15.139:0/1745672499 learned_addr learned my addr 172.21.15.139:0/1745672499 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:44.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.554+0000 7fa34a59c700 1 --2- 172.21.15.139:0/1745672499 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa34c0f6670 0x7fa34c106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:44.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.554+0000 7fa349d9b700 1 -- 172.21.15.139:0/1745672499 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa34c0f6670 msgr2=0x7fa34c106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:44.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.554+0000 7fa349d9b700 1 --2- 172.21.15.139:0/1745672499 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa34c0f6670 0x7fa34c106ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:44.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.554+0000 7fa349d9b700 1 -- 172.21.15.139:0/1745672499 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c005040 con 0x7fa34c0f7910 2021-11-20T06:06:44.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.554+0000 7fa34a59c700 1 --2- 172.21.15.139:0/1745672499 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa34c0f6670 0x7fa34c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:06:44.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.554+0000 7fa349d9b700 1 --2- 172.21.15.139:0/1745672499 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c0f7910 0x7fa34c1073b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fa3400026d0 tx=0x7fa340009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:44.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.554+0000 7fa33affd700 1 -- 172.21.15.139:0/1745672499 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa340009320 con 0x7fa34c0f7910 2021-11-20T06:06:44.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.554+0000 7fa350be5700 1 -- 172.21.15.139:0/1745672499 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34c10c540 con 0x7fa34c0f7910 2021-11-20T06:06:44.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.554+0000 7fa33affd700 1 -- 172.21.15.139:0/1745672499 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa34000f040 con 0x7fa34c0f7910 2021-11-20T06:06:44.556 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.554+0000 7fa33affd700 1 -- 172.21.15.139:0/1745672499 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3400147e0 con 0x7fa34c0f7910 2021-11-20T06:06:44.556 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.554+0000 7fa350be5700 1 -- 172.21.15.139:0/1745672499 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34c10cb00 con 0x7fa34c0f7910 2021-11-20T06:06:44.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.555+0000 7fa350be5700 1 -- 172.21.15.139:0/1745672499 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34c027a00 con 0x7fa34c0f7910 2021-11-20T06:06:44.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.555+0000 7fa33affd700 1 -- 172.21.15.139:0/1745672499 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa340009480 con 0x7fa34c0f7910 2021-11-20T06:06:44.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.556+0000 7fa33affd700 1 --2- 172.21.15.139:0/1745672499 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa33405c2a0 0x7fa33405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:44.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.556+0000 7fa33affd700 1 -- 172.21.15.139:0/1745672499 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa3400769b0 con 0x7fa34c0f7910 2021-11-20T06:06:44.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.556+0000 7fa34a59c700 1 --2- 172.21.15.139:0/1745672499 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa33405c2a0 0x7fa33405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:44.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.556+0000 7fa34a59c700 1 --2- 172.21.15.139:0/1745672499 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa33405c2a0 0x7fa33405e760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa33c004e30 tx=0x7fa33c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:44.560 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.560+0000 7fa33affd700 1 -- 172.21.15.139:0/1745672499 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3400492c0 con 0x7fa34c0f7910 2021-11-20T06:06:44.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:44 smithi178 conmon[36600]: audit 2021-11-20T06:06:43.417728+0000 mon.smithi139 (mon.0) 741 : audit [DBG] from='client.? 172.21.15.139:0/2715593366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:44.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:44 smithi178 conmon[36600]: cluster 2021-11-20T06:06:44.236901 2021-11-20T06:06:44.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:44 smithi178 conmon[36600]: +0000 mon.smithi139 (mon.0) 742 : cluster [DBG] mds.? [v2:172.21.15.139:6826/811251312,v1:172.21.15.139:6827/811251312] up:rejoin 2021-11-20T06:06:44.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:44 smithi178 conmon[36600]: cluster 2021-11-20T06:06:44.236969+0000 mon.smithi139 (mon.0) 743 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi139.riczqb=up:rejoin} 2 up:standby 2021-11-20T06:06:44.733 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.748+0000 7fa350be5700 1 -- 172.21.15.139:0/1745672499 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa34c0fee00 con 0x7fa34c0f7910 2021-11-20T06:06:44.734 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.749+0000 7fa33affd700 1 -- 172.21.15.139:0/1745672499 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2562 (secure 0 0 0) 0x7fa34004c8e0 con 0x7fa34c0f7910 2021-11-20T06:06:44.735 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:44.735 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":64,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:44.233143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14660":{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:44.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.750+0000 7fa350be5700 1 -- 172.21.15.139:0/1745672499 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa33405c2a0 msgr2=0x7fa33405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:44.738 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.750+0000 7fa350be5700 1 --2- 172.21.15.139:0/1745672499 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa33405c2a0 0x7fa33405e760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa33c004e30 tx=0x7fa33c009b30).stop 2021-11-20T06:06:44.738 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.750+0000 7fa350be5700 1 -- 172.21.15.139:0/1745672499 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c0f7910 msgr2=0x7fa34c1073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:44.738 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.750+0000 7fa350be5700 1 --2- 172.21.15.139:0/1745672499 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c0f7910 0x7fa34c1073b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fa3400026d0 tx=0x7fa340009de0).stop 2021-11-20T06:06:44.739 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.750+0000 7fa350be5700 1 -- 172.21.15.139:0/1745672499 shutdown_connections 2021-11-20T06:06:44.739 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.750+0000 7fa350be5700 1 --2- 172.21.15.139:0/1745672499 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa33405c2a0 0x7fa33405e760 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:44.739 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.750+0000 7fa350be5700 1 --2- 172.21.15.139:0/1745672499 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa34c0f6670 0x7fa34c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:44.739 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.750+0000 7fa350be5700 1 --2- 172.21.15.139:0/1745672499 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa34c0f7910 0x7fa34c1073b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:44.739 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.750+0000 7fa350be5700 1 -- 172.21.15.139:0/1745672499 >> 172.21.15.139:0/1745672499 conn(0x7fa34c0f1db0 msgr2=0x7fa34c0fb360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:44.740 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.751+0000 7fa350be5700 1 -- 172.21.15.139:0/1745672499 shutdown_connections 2021-11-20T06:06:44.740 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:44.751+0000 7fa350be5700 1 -- 172.21.15.139:0/1745672499 wait complete. 2021-11-20T06:06:44.740 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 64 2021-11-20T06:06:44.747 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:44.233143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14660': {'gid': 14660, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6827/811251312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 811251312}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 811251312}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:44.748 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14660:up:rejoin 2021-11-20T06:06:45.460 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:45 smithi139 conmon[33501]: cluster 2021-11-20T06:06:44.261288+0000 mgr.smithi139.dplyxn (mgr.14162) 288 : cluster [DBG] pgmap v260: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T06:06:45.460 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:45 smithi139 conmon[33501]: audit 2021-11-20T06:06:44.749594+0000 mon.smithi139 (mon.0) 744 : audit [DBG] from='client.? 172.21.15.139:0/1745672499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:45.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:45 smithi178 conmon[36600]: cluster 2021-11-20T06:06:44.261288+0000 mgr.smithi139.dplyxn (mgr.14162) 288 : cluster [DBG] pgmap v260: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T06:06:45.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:45 smithi178 conmon[36600]: audit 2021-11-20T06:06:44.749594+0000 mon.smithi139 (mon.0) 744 : audit [DBG] from='client.? 172.21.15.139:0/1745672499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:45.749 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:45.870 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.883+0000 7fc6df2b5700 1 -- 172.21.15.139:0/2436408142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6d80f9800 msgr2=0x7fc6d80fbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:45.870 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.883+0000 7fc6df2b5700 1 --2- 172.21.15.139:0/2436408142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6d80f9800 0x7fc6d80fbc50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fc6c8004660 tx=0x7fc6c8009b30).stop 2021-11-20T06:06:45.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.883+0000 7fc6df2b5700 1 -- 172.21.15.139:0/2436408142 shutdown_connections 2021-11-20T06:06:45.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.883+0000 7fc6df2b5700 1 --2- 172.21.15.139:0/2436408142 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6d80f9800 0x7fc6d80fbc50 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:45.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.883+0000 7fc6df2b5700 1 --2- 172.21.15.139:0/2436408142 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d8060140 0x7fc6d80f9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:45.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.883+0000 7fc6df2b5700 1 -- 172.21.15.139:0/2436408142 >> 172.21.15.139:0/2436408142 conn(0x7fc6d80f0ce0 msgr2=0x7fc6d80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:45.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.884+0000 7fc6df2b5700 1 -- 172.21.15.139:0/2436408142 shutdown_connections 2021-11-20T06:06:45.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.884+0000 7fc6df2b5700 1 -- 172.21.15.139:0/2436408142 wait complete. 2021-11-20T06:06:45.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.884+0000 7fc6df2b5700 1 Processor -- start 2021-11-20T06:06:45.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.884+0000 7fc6df2b5700 1 -- start start 2021-11-20T06:06:45.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.884+0000 7fc6df2b5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6d8060140 0x7fc6d8106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:45.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.884+0000 7fc6df2b5700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d80f9800 0x7fc6d8107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:45.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.884+0000 7fc6df2b5700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d8107930 con 0x7fc6d8060140 2021-11-20T06:06:45.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.884+0000 7fc6df2b5700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d8107a70 con 0x7fc6d80f9800 2021-11-20T06:06:45.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.884+0000 7fc6dd051700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6d8060140 0x7fc6d8106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:45.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.884+0000 7fc6dd051700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6d8060140 0x7fc6d8106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58438/0 (socket says 172.21.15.139:58438) 2021-11-20T06:06:45.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.884+0000 7fc6dd051700 1 -- 172.21.15.139:0/3866399212 learned_addr learned my addr 172.21.15.139:0/3866399212 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:45.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.885+0000 7fc6dd051700 1 -- 172.21.15.139:0/3866399212 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d80f9800 msgr2=0x7fc6d8107380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:45.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.885+0000 7fc6dd051700 1 --2- 172.21.15.139:0/3866399212 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d80f9800 0x7fc6d8107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:45.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.885+0000 7fc6dd051700 1 -- 172.21.15.139:0/3866399212 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c8005040 con 0x7fc6d8060140 2021-11-20T06:06:45.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.885+0000 7fc6dd051700 1 --2- 172.21.15.139:0/3866399212 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6d8060140 0x7fc6d8106e90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fc6d000c7c0 tx=0x7fc6d0004d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:45.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.885+0000 7fc6cdffb700 1 -- 172.21.15.139:0/3866399212 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6d000a6e0 con 0x7fc6d8060140 2021-11-20T06:06:45.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.885+0000 7fc6cdffb700 1 -- 172.21.15.139:0/3866399212 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6d000acf0 con 0x7fc6d8060140 2021-11-20T06:06:45.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.885+0000 7fc6df2b5700 1 -- 172.21.15.139:0/3866399212 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d810c510 con 0x7fc6d8060140 2021-11-20T06:06:45.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.885+0000 7fc6cdffb700 1 -- 172.21.15.139:0/3866399212 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6d0012df0 con 0x7fc6d8060140 2021-11-20T06:06:45.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.885+0000 7fc6df2b5700 1 -- 172.21.15.139:0/3866399212 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d810cad0 con 0x7fc6d8060140 2021-11-20T06:06:45.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.886+0000 7fc6df2b5700 1 -- 172.21.15.139:0/3866399212 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d8027a00 con 0x7fc6d8060140 2021-11-20T06:06:45.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.886+0000 7fc6cdffb700 1 -- 172.21.15.139:0/3866399212 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc6d0019030 con 0x7fc6d8060140 2021-11-20T06:06:45.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.887+0000 7fc6cdffb700 1 --2- 172.21.15.139:0/3866399212 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc6c405c2a0 0x7fc6c405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:45.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.887+0000 7fc6cdffb700 1 -- 172.21.15.139:0/3866399212 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc6d0076400 con 0x7fc6d8060140 2021-11-20T06:06:45.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.887+0000 7fc6dc850700 1 --2- 172.21.15.139:0/3866399212 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc6c405c2a0 0x7fc6c405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:45.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.887+0000 7fc6dc850700 1 --2- 172.21.15.139:0/3866399212 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc6c405c2a0 0x7fc6c405e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fc6c8009f50 tx=0x7fc6c8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:45.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:45.890+0000 7fc6cdffb700 1 -- 172.21.15.139:0/3866399212 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc6d00462b0 con 0x7fc6d8060140 2021-11-20T06:06:46.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:46.081+0000 7fc6df2b5700 1 -- 172.21.15.139:0/3866399212 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6d80fee30 con 0x7fc6d8060140 2021-11-20T06:06:46.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:46.082+0000 7fc6cdffb700 1 -- 172.21.15.139:0/3866399212 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2562 (secure 0 0 0) 0x7fc6d0017070 con 0x7fc6d8060140 2021-11-20T06:06:46.067 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:46.067 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":64,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:44.233143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14660":{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:46.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:46.083+0000 7fc6df2b5700 1 -- 172.21.15.139:0/3866399212 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc6c405c2a0 msgr2=0x7fc6c405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:46.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:46.083+0000 7fc6df2b5700 1 --2- 172.21.15.139:0/3866399212 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc6c405c2a0 0x7fc6c405e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fc6c8009f50 tx=0x7fc6c8009b30).stop 2021-11-20T06:06:46.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:46.083+0000 7fc6df2b5700 1 -- 172.21.15.139:0/3866399212 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6d8060140 msgr2=0x7fc6d8106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:46.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:46.083+0000 7fc6df2b5700 1 --2- 172.21.15.139:0/3866399212 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6d8060140 0x7fc6d8106e90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fc6d000c7c0 tx=0x7fc6d0004d70).stop 2021-11-20T06:06:46.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:46.083+0000 7fc6df2b5700 1 -- 172.21.15.139:0/3866399212 shutdown_connections 2021-11-20T06:06:46.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:46.083+0000 7fc6df2b5700 1 --2- 172.21.15.139:0/3866399212 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc6c405c2a0 0x7fc6c405e760 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:46.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:46.083+0000 7fc6df2b5700 1 --2- 172.21.15.139:0/3866399212 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6d8060140 0x7fc6d8106e90 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:46.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:46.083+0000 7fc6df2b5700 1 --2- 172.21.15.139:0/3866399212 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6d80f9800 0x7fc6d8107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:46.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:46.083+0000 7fc6df2b5700 1 -- 172.21.15.139:0/3866399212 >> 172.21.15.139:0/3866399212 conn(0x7fc6d80f0ce0 msgr2=0x7fc6d80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:46.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:46.083+0000 7fc6df2b5700 1 -- 172.21.15.139:0/3866399212 shutdown_connections 2021-11-20T06:06:46.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:46.083+0000 7fc6df2b5700 1 -- 172.21.15.139:0/3866399212 wait complete. 2021-11-20T06:06:46.071 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 64 2021-11-20T06:06:46.080 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:44.233143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14660': {'gid': 14660, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6827/811251312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 811251312}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 811251312}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:46.080 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14660:up:rejoin 2021-11-20T06:06:46.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:46 smithi139 conmon[33501]: audit 2021-11-20T06:06:46.082384+0000 mon.smithi139 (mon 2021-11-20T06:06:46.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:46 smithi139 conmon[33501]: .0) 745 : audit [DBG] from='client.? 172.21.15.139:0/3866399212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:46.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:46 smithi178 conmon[36600]: audit 2021-11-20T06:06:46.082384+0000 mon.smithi139 (mon.0) 745 : audit [DBG] 2021-11-20T06:06:46.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:46 smithi178 conmon[36600]: from='client.? 172.21.15.139:0/3866399212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:47.082 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:47.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.233+0000 7ff626f71700 1 -- 172.21.15.139:0/1276669404 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6200f55a0 msgr2=0x7ff6200f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:47.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.233+0000 7ff626f71700 1 --2- 172.21.15.139:0/1276669404 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6200f55a0 0x7ff6200f59c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7ff618004660 tx=0x7ff618009b30).stop 2021-11-20T06:06:47.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.233+0000 7ff626f71700 1 -- 172.21.15.139:0/1276669404 shutdown_connections 2021-11-20T06:06:47.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.233+0000 7ff626f71700 1 --2- 172.21.15.139:0/1276669404 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6200f6840 0x7ff6200f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:47.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.233+0000 7ff626f71700 1 --2- 172.21.15.139:0/1276669404 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6200f55a0 0x7ff6200f59c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:47.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.233+0000 7ff626f71700 1 -- 172.21.15.139:0/1276669404 >> 172.21.15.139:0/1276669404 conn(0x7ff6200f0ce0 msgr2=0x7ff6200f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:47.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.233+0000 7ff626f71700 1 -- 172.21.15.139:0/1276669404 shutdown_connections 2021-11-20T06:06:47.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.233+0000 7ff626f71700 1 -- 172.21.15.139:0/1276669404 wait complete. 2021-11-20T06:06:47.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.233+0000 7ff626f71700 1 Processor -- start 2021-11-20T06:06:47.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.233+0000 7ff626f71700 1 -- start start 2021-11-20T06:06:47.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.234+0000 7ff626f71700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6200f55a0 0x7ff62005ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:47.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.234+0000 7ff626f71700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6200f6840 0x7ff62010bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:47.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.234+0000 7ff626f71700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62010c180 con 0x7ff6200f55a0 2021-11-20T06:06:47.223 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.234+0000 7ff626f71700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62010c2c0 con 0x7ff6200f6840 2021-11-20T06:06:47.223 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.234+0000 7ff624d0d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6200f55a0 0x7ff62005ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:47.223 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.234+0000 7ff624d0d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6200f55a0 0x7ff62005ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58450/0 (socket says 172.21.15.139:58450) 2021-11-20T06:06:47.223 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.234+0000 7ff624d0d700 1 -- 172.21.15.139:0/2221147707 learned_addr learned my addr 172.21.15.139:0/2221147707 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:47.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.234+0000 7ff61ffff700 1 --2- 172.21.15.139:0/2221147707 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6200f6840 0x7ff62010bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:47.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.234+0000 7ff624d0d700 1 -- 172.21.15.139:0/2221147707 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6200f6840 msgr2=0x7ff62010bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:47.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.234+0000 7ff624d0d700 1 --2- 172.21.15.139:0/2221147707 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6200f6840 0x7ff62010bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:47.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.234+0000 7ff624d0d700 1 -- 172.21.15.139:0/2221147707 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff618005040 con 0x7ff6200f55a0 2021-11-20T06:06:47.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.234+0000 7ff624d0d700 1 --2- 172.21.15.139:0/2221147707 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6200f55a0 0x7ff62005ff20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7ff618009f50 tx=0x7ff618009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:47.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.234+0000 7ff61d7fa700 1 -- 172.21.15.139:0/2221147707 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff618018070 con 0x7ff6200f55a0 2021-11-20T06:06:47.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.235+0000 7ff626f71700 1 -- 172.21.15.139:0/2221147707 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff62010c4c0 con 0x7ff6200f55a0 2021-11-20T06:06:47.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.235+0000 7ff61d7fa700 1 -- 172.21.15.139:0/2221147707 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff618013410 con 0x7ff6200f55a0 2021-11-20T06:06:47.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.235+0000 7ff61d7fa700 1 -- 172.21.15.139:0/2221147707 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff61801de20 con 0x7ff6200f55a0 2021-11-20T06:06:47.226 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.235+0000 7ff626f71700 1 -- 172.21.15.139:0/2221147707 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62010ca20 con 0x7ff6200f55a0 2021-11-20T06:06:47.227 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.236+0000 7ff61d7fa700 1 -- 172.21.15.139:0/2221147707 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff61801d420 con 0x7ff6200f55a0 2021-11-20T06:06:47.227 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.236+0000 7ff626f71700 1 -- 172.21.15.139:0/2221147707 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff620027a00 con 0x7ff6200f55a0 2021-11-20T06:06:47.227 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.236+0000 7ff61d7fa700 1 --2- 172.21.15.139:0/2221147707 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff60c05c2a0 0x7ff60c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:47.228 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.236+0000 7ff61d7fa700 1 -- 172.21.15.139:0/2221147707 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff618077ec0 con 0x7ff6200f55a0 2021-11-20T06:06:47.228 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.236+0000 7ff61ffff700 1 --2- 172.21.15.139:0/2221147707 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff60c05c2a0 0x7ff60c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:47.228 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.237+0000 7ff61ffff700 1 --2- 172.21.15.139:0/2221147707 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff60c05c2a0 0x7ff60c05e760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7ff610007db0 tx=0x7ff610009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:47.228 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.241+0000 7ff61d7fa700 1 -- 172.21.15.139:0/2221147707 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff61804a7d0 con 0x7ff6200f55a0 2021-11-20T06:06:47.420 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.435+0000 7ff626f71700 1 -- 172.21.15.139:0/2221147707 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6200fdd30 con 0x7ff6200f55a0 2021-11-20T06:06:47.421 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.437+0000 7ff61d7fa700 1 -- 172.21.15.139:0/2221147707 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2562 (secure 0 0 0) 0x7ff61804ddf0 con 0x7ff6200f55a0 2021-11-20T06:06:47.422 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:47.422 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":64,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:44.233143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14660":{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:47.423 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.438+0000 7ff626f71700 1 -- 172.21.15.139:0/2221147707 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff60c05c2a0 msgr2=0x7ff60c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:47.423 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.438+0000 7ff626f71700 1 --2- 172.21.15.139:0/2221147707 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff60c05c2a0 0x7ff60c05e760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7ff610007db0 tx=0x7ff610009040).stop 2021-11-20T06:06:47.424 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.438+0000 7ff626f71700 1 -- 172.21.15.139:0/2221147707 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6200f55a0 msgr2=0x7ff62005ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:47.424 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.438+0000 7ff626f71700 1 --2- 172.21.15.139:0/2221147707 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6200f55a0 0x7ff62005ff20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7ff618009f50 tx=0x7ff618009b30).stop 2021-11-20T06:06:47.424 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.438+0000 7ff626f71700 1 -- 172.21.15.139:0/2221147707 shutdown_connections 2021-11-20T06:06:47.424 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.438+0000 7ff626f71700 1 --2- 172.21.15.139:0/2221147707 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff60c05c2a0 0x7ff60c05e760 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:47.424 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.438+0000 7ff626f71700 1 --2- 172.21.15.139:0/2221147707 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6200f55a0 0x7ff62005ff20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:47.425 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.438+0000 7ff626f71700 1 --2- 172.21.15.139:0/2221147707 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6200f6840 0x7ff62010bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:47.425 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.438+0000 7ff626f71700 1 -- 172.21.15.139:0/2221147707 >> 172.21.15.139:0/2221147707 conn(0x7ff6200f0ce0 msgr2=0x7ff6200fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:47.425 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.439+0000 7ff626f71700 1 -- 172.21.15.139:0/2221147707 shutdown_connections 2021-11-20T06:06:47.425 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:47.439+0000 7ff626f71700 1 -- 172.21.15.139:0/2221147707 wait complete. 2021-11-20T06:06:47.426 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 64 2021-11-20T06:06:47.434 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:47 smithi139 conmon[33501]: cluster 2021-11-20T06:06:46.261711+0000 2021-11-20T06:06:47.435 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:47 smithi139 conmon[33501]: mgr.smithi139.dplyxn (mgr.14162) 289 : cluster [DBG] pgmap v261: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2021-11-20T06:06:47.435 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:47 smithi139 conmon[33501]: audit 2021-11-20T06:06:47.184059+0000 mon.smithi139 (mon.0) 2021-11-20T06:06:47.435 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:47 smithi139 conmon[33501]: 746 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:06:47.436 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:44.233143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14660': {'gid': 14660, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6827/811251312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 811251312}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 811251312}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:47.437 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14660:up:rejoin 2021-11-20T06:06:47.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:47 smithi178 conmon[36600]: cluster 2021-11-20T06:06:46.261711+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:06:47.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:47 smithi178 conmon[36600]: ) 289 : cluster [DBG] pgmap v261: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2021-11-20T06:06:47.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:47 smithi178 conmon[36600]: audit 2021-11-20T06:06:47.184059+0000 mon.smithi139 (mon.0) 746 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:06:48.438 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:48.568 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.581+0000 7f38c7b0b700 1 -- 172.21.15.139:0/434419736 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38c00f6820 msgr2=0x7f38c00f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:48.568 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.581+0000 7f38c7b0b700 1 --2- 172.21.15.139:0/434419736 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38c00f6820 0x7f38c00f6c80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f38b4004660 tx=0x7f38b4009b30).stop 2021-11-20T06:06:48.568 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.581+0000 7f38c7b0b700 1 -- 172.21.15.139:0/434419736 shutdown_connections 2021-11-20T06:06:48.568 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.581+0000 7f38c7b0b700 1 --2- 172.21.15.139:0/434419736 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38c00f6820 0x7f38c00f6c80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:48.569 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.581+0000 7f38c7b0b700 1 --2- 172.21.15.139:0/434419736 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38c00f5580 0x7f38c00f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:48.569 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.581+0000 7f38c7b0b700 1 -- 172.21.15.139:0/434419736 >> 172.21.15.139:0/434419736 conn(0x7f38c00f0ce0 msgr2=0x7f38c00f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:48.570 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.581+0000 7f38c7b0b700 1 -- 172.21.15.139:0/434419736 shutdown_connections 2021-11-20T06:06:48.570 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.581+0000 7f38c7b0b700 1 -- 172.21.15.139:0/434419736 wait complete. 2021-11-20T06:06:48.571 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.581+0000 7f38c7b0b700 1 Processor -- start 2021-11-20T06:06:48.571 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.581+0000 7f38c7b0b700 1 -- start start 2021-11-20T06:06:48.571 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38c7b0b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38c00f5580 0x7f38c0060550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:48.572 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38c7b0b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38c00f6820 0x7f38c010bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:48.572 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38c7b0b700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c010c1a0 con 0x7f38c00f6820 2021-11-20T06:06:48.572 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38c7b0b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c010c2e0 con 0x7f38c00f5580 2021-11-20T06:06:48.572 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38c50a6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38c00f6820 0x7f38c010bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:48.573 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38c50a6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38c00f6820 0x7f38c010bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58462/0 (socket says 172.21.15.139:58462) 2021-11-20T06:06:48.573 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38c50a6700 1 -- 172.21.15.139:0/3839085246 learned_addr learned my addr 172.21.15.139:0/3839085246 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:48.573 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38c58a7700 1 --2- 172.21.15.139:0/3839085246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38c00f5580 0x7f38c0060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:48.573 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38c50a6700 1 -- 172.21.15.139:0/3839085246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38c00f5580 msgr2=0x7f38c0060550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:48.573 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38c50a6700 1 --2- 172.21.15.139:0/3839085246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38c00f5580 0x7f38c0060550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:48.574 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38c50a6700 1 -- 172.21.15.139:0/3839085246 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b4005040 con 0x7f38c00f6820 2021-11-20T06:06:48.574 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38c50a6700 1 --2- 172.21.15.139:0/3839085246 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38c00f6820 0x7f38c010bcb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f38b4009830 tx=0x7f38b4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:48.574 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38be7fc700 1 -- 172.21.15.139:0/3839085246 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38b4018070 con 0x7f38c00f6820 2021-11-20T06:06:48.574 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38c7b0b700 1 -- 172.21.15.139:0/3839085246 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c010c4e0 con 0x7f38c00f6820 2021-11-20T06:06:48.575 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38be7fc700 1 -- 172.21.15.139:0/3839085246 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38b4013410 con 0x7f38c00f6820 2021-11-20T06:06:48.575 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38be7fc700 1 -- 172.21.15.139:0/3839085246 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38b401c3c0 con 0x7f38c00f6820 2021-11-20T06:06:48.575 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.582+0000 7f38c7b0b700 1 -- 172.21.15.139:0/3839085246 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c010ca40 con 0x7f38c00f6820 2021-11-20T06:06:48.576 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.583+0000 7f38c7b0b700 1 -- 172.21.15.139:0/3839085246 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c0027a00 con 0x7f38c00f6820 2021-11-20T06:06:48.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.584+0000 7f38be7fc700 1 -- 172.21.15.139:0/3839085246 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f38b401d420 con 0x7f38c00f6820 2021-11-20T06:06:48.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.584+0000 7f38be7fc700 1 --2- 172.21.15.139:0/3839085246 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f38a805c2a0 0x7f38a805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:48.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.584+0000 7f38be7fc700 1 -- 172.21.15.139:0/3839085246 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f38b4077ef0 con 0x7f38c00f6820 2021-11-20T06:06:48.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.584+0000 7f38c58a7700 1 --2- 172.21.15.139:0/3839085246 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f38a805c2a0 0x7f38a805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:48.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.584+0000 7f38c58a7700 1 --2- 172.21.15.139:0/3839085246 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f38a805c2a0 0x7f38a805e760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f38c00f6680 tx=0x7f38b0009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:48.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.588+0000 7f38be7fc700 1 -- 172.21.15.139:0/3839085246 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38b404a800 con 0x7f38c00f6820 2021-11-20T06:06:48.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:48 smithi178 conmon[36600]: audit 2021-11-20T06:06:47.437347+0000 mon.smithi139 (mon.0) 747 : audit [DBG] from='client.? 172.21.15.139:0/2221147707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:48.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:48 smithi178 conmon[36600]: audit 2021-11-20T06:06:47.567420+0000 mon.smithi139 (mon.0) 748 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:06:48.755 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.771+0000 7f38c7b0b700 1 -- 172.21.15.139:0/3839085246 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38c00fdd10 con 0x7f38c00f6820 2021-11-20T06:06:48.756 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.771+0000 7f38be7fc700 1 -- 172.21.15.139:0/3839085246 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2562 (secure 0 0 0) 0x7f38b404de20 con 0x7f38c00f6820 2021-11-20T06:06:48.756 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:48.757 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":64,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:44.233143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14660":{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:48.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.773+0000 7f38c7b0b700 1 -- 172.21.15.139:0/3839085246 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f38a805c2a0 msgr2=0x7f38a805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:48.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.773+0000 7f38c7b0b700 1 --2- 172.21.15.139:0/3839085246 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f38a805c2a0 0x7f38a805e760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f38c00f6680 tx=0x7f38b0009040).stop 2021-11-20T06:06:48.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.773+0000 7f38c7b0b700 1 -- 172.21.15.139:0/3839085246 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38c00f6820 msgr2=0x7f38c010bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:48.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.773+0000 7f38c7b0b700 1 --2- 172.21.15.139:0/3839085246 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38c00f6820 0x7f38c010bcb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f38b4009830 tx=0x7f38b4009b30).stop 2021-11-20T06:06:48.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.773+0000 7f38c7b0b700 1 -- 172.21.15.139:0/3839085246 shutdown_connections 2021-11-20T06:06:48.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.773+0000 7f38c7b0b700 1 --2- 172.21.15.139:0/3839085246 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f38a805c2a0 0x7f38a805e760 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:48.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.773+0000 7f38c7b0b700 1 --2- 172.21.15.139:0/3839085246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38c00f5580 0x7f38c0060550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:48.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.773+0000 7f38c7b0b700 1 --2- 172.21.15.139:0/3839085246 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f38c00f6820 0x7f38c010bcb0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:48.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.773+0000 7f38c7b0b700 1 -- 172.21.15.139:0/3839085246 >> 172.21.15.139:0/3839085246 conn(0x7f38c00f0ce0 msgr2=0x7f38c00fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:48.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.773+0000 7f38c7b0b700 1 -- 172.21.15.139:0/3839085246 shutdown_connections 2021-11-20T06:06:48.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:48.773+0000 7f38c7b0b700 1 -- 172.21.15.139:0/3839085246 wait complete. 2021-11-20T06:06:48.761 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 64 2021-11-20T06:06:48.768 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:48 smithi139 conmon[33501]: audit 2021-11-20T06:06:47.437347+0000 mon.smithi139 (mon.0) 2021-11-20T06:06:48.768 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:48 smithi139 conmon[33501]: 747 : audit [DBG] from='client.? 172.21.15.139:0/2221147707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:48.769 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:48 smithi139 conmon[33501]: audit 2021-11-20T06:06:47.567420+0000 mon.smithi139 (mon.0) 748 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:06:48.770 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:44.233143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14660': {'gid': 14660, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6827/811251312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 811251312}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 811251312}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:48.770 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14660:up:rejoin 2021-11-20T06:06:49.771 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:49.900 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.913+0000 7fae28794700 1 -- 172.21.15.139:0/1565859724 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae200f6b90 msgr2=0x7fae200f6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:49.900 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.913+0000 7fae28794700 1 --2- 172.21.15.139:0/1565859724 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae200f6b90 0x7fae200f6fb0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fae1c0099e0 tx=0x7fae1c004d10).stop 2021-11-20T06:06:49.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.913+0000 7fae28794700 1 -- 172.21.15.139:0/1565859724 shutdown_connections 2021-11-20T06:06:49.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.913+0000 7fae28794700 1 --2- 172.21.15.139:0/1565859724 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae200f5890 0x7fae200f5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:49.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.913+0000 7fae28794700 1 --2- 172.21.15.139:0/1565859724 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae200f6b90 0x7fae200f6fb0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:49.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.913+0000 7fae28794700 1 -- 172.21.15.139:0/1565859724 >> 172.21.15.139:0/1565859724 conn(0x7fae200f04a0 msgr2=0x7fae200f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:49.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.913+0000 7fae28794700 1 -- 172.21.15.139:0/1565859724 shutdown_connections 2021-11-20T06:06:49.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.913+0000 7fae28794700 1 -- 172.21.15.139:0/1565859724 wait complete. 2021-11-20T06:06:49.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae28794700 1 Processor -- start 2021-11-20T06:06:49.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae28794700 1 -- start start 2021-11-20T06:06:49.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae28794700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae200f5890 0x7fae200f5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:49.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae28794700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae200f6b90 0x7fae2010e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:49.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae28794700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae20107bc0 con 0x7fae200f5890 2021-11-20T06:06:49.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae28794700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2010e6d0 con 0x7fae200f6b90 2021-11-20T06:06:49.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae26530700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae200f5890 0x7fae200f5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:49.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae26530700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae200f5890 0x7fae200f5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58472/0 (socket says 172.21.15.139:58472) 2021-11-20T06:06:49.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae26530700 1 -- 172.21.15.139:0/1999663696 learned_addr learned my addr 172.21.15.139:0/1999663696 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:49.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae26530700 1 -- 172.21.15.139:0/1999663696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae200f6b90 msgr2=0x7fae2010e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:49.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae26530700 1 --2- 172.21.15.139:0/1999663696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae200f6b90 0x7fae2010e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:49.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae26530700 1 -- 172.21.15.139:0/1999663696 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c009640 con 0x7fae200f5890 2021-11-20T06:06:49.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae26530700 1 --2- 172.21.15.139:0/1999663696 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae200f5890 0x7fae200f5230 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fae1c007830 tx=0x7fae1c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:49.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae16ffd700 1 -- 172.21.15.139:0/1999663696 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae1c00e030 con 0x7fae200f5890 2021-11-20T06:06:49.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae16ffd700 1 -- 172.21.15.139:0/1999663696 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae1c01d8d0 con 0x7fae200f5890 2021-11-20T06:06:49.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae28794700 1 -- 172.21.15.139:0/1999663696 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2010e810 con 0x7fae200f5890 2021-11-20T06:06:49.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.914+0000 7fae16ffd700 1 -- 172.21.15.139:0/1999663696 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae1c018070 con 0x7fae200f5890 2021-11-20T06:06:49.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.915+0000 7fae28794700 1 -- 172.21.15.139:0/1999663696 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2010ecc0 con 0x7fae200f5890 2021-11-20T06:06:49.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.915+0000 7fae28794700 1 -- 172.21.15.139:0/1999663696 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae20027a00 con 0x7fae200f5890 2021-11-20T06:06:49.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.916+0000 7fae16ffd700 1 -- 172.21.15.139:0/1999663696 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fae1c01da40 con 0x7fae200f5890 2021-11-20T06:06:49.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.916+0000 7fae16ffd700 1 --2- 172.21.15.139:0/1999663696 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fae0c05c2a0 0x7fae0c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:49.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.916+0000 7fae16ffd700 1 -- 172.21.15.139:0/1999663696 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fae1c0780d0 con 0x7fae200f5890 2021-11-20T06:06:49.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.916+0000 7fae25d2f700 1 --2- 172.21.15.139:0/1999663696 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fae0c05c2a0 0x7fae0c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:49.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.916+0000 7fae25d2f700 1 --2- 172.21.15.139:0/1999663696 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fae0c05c2a0 0x7fae0c05e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fae200f6960 tx=0x7fae10007a60).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:49.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:49.920+0000 7fae16ffd700 1 -- 172.21.15.139:0/1999663696 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae1c04aa60 con 0x7fae200f5890 2021-11-20T06:06:49.941 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:49 smithi178 conmon[36600]: cluster 2021-11-20T06:06:48.262106+0000 mgr.smithi139.dplyxn (mgr.14162) 290 : cluster [DBG] pgmap v262: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:49.941 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:49 smithi178 conmon[36600]: audit 2021-11-20T06:06:48.772060+0000 mon.smithi139 (mon.0) 749 : audit [DBG] from='client.? 172.21.15.139:0/3839085246' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:50.027 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:49 smithi139 conmon[33501]: cluster 2021-11-20T06:06:48.262106+0000 mgr.smithi139.dplyxn (mgr.14162) 290 : 2021-11-20T06:06:50.027 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:49 smithi139 conmon[33501]: cluster [DBG] pgmap v262: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:50.028 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:49 smithi139 conmon[33501]: audit 2021-11-20T06:06:48.772060+0000 mon.smithi139 (mon.0) 749 : audit [DBG] from='client.? 172.21.15.139:0/3839085246' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:50.088 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:50.103+0000 7fae28794700 1 -- 172.21.15.139:0/1999663696 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae200fdd80 con 0x7fae200f5890 2021-11-20T06:06:50.089 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:50.104+0000 7fae16ffd700 1 -- 172.21.15.139:0/1999663696 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2562 (secure 0 0 0) 0x7fae1c04e080 con 0x7fae200f5890 2021-11-20T06:06:50.089 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:50.090 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":64,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:44.233143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14660":{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:50.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:50.105+0000 7fae28794700 1 -- 172.21.15.139:0/1999663696 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fae0c05c2a0 msgr2=0x7fae0c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:50.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:50.105+0000 7fae28794700 1 --2- 172.21.15.139:0/1999663696 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fae0c05c2a0 0x7fae0c05e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fae200f6960 tx=0x7fae10007a60).stop 2021-11-20T06:06:50.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:50.105+0000 7fae28794700 1 -- 172.21.15.139:0/1999663696 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae200f5890 msgr2=0x7fae200f5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:50.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:50.105+0000 7fae28794700 1 --2- 172.21.15.139:0/1999663696 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae200f5890 0x7fae200f5230 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fae1c007830 tx=0x7fae1c004d10).stop 2021-11-20T06:06:50.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:50.105+0000 7fae28794700 1 -- 172.21.15.139:0/1999663696 shutdown_connections 2021-11-20T06:06:50.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:50.105+0000 7fae28794700 1 --2- 172.21.15.139:0/1999663696 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fae0c05c2a0 0x7fae0c05e760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:50.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:50.105+0000 7fae28794700 1 --2- 172.21.15.139:0/1999663696 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fae200f5890 0x7fae200f5230 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:50.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:50.105+0000 7fae28794700 1 --2- 172.21.15.139:0/1999663696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae200f6b90 0x7fae2010e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:50.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:50.105+0000 7fae28794700 1 -- 172.21.15.139:0/1999663696 >> 172.21.15.139:0/1999663696 conn(0x7fae200f04a0 msgr2=0x7fae200fa2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:50.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:50.105+0000 7fae28794700 1 -- 172.21.15.139:0/1999663696 shutdown_connections 2021-11-20T06:06:50.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:50.105+0000 7fae28794700 1 -- 172.21.15.139:0/1999663696 wait complete. 2021-11-20T06:06:50.096 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 64 2021-11-20T06:06:50.102 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:44.233143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14660': {'gid': 14660, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6827/811251312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 811251312}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 811251312}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:50.103 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14660:up:rejoin 2021-11-20T06:06:50.850 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:50 smithi139 conmon[33501]: audit 2021-11-20T06:06:50.104478+0000 mon.smithi139 (mon. 2021-11-20T06:06:50.850 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:50 smithi139 conmon[33501]: 0) 750 : audit [DBG] from='client.? 172.21.15.139:0/1999663696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:50.941 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:50 smithi178 conmon[36600]: audit 2021-11-20T06:06:50.104478+0000 mon.smithi139 (mon.0) 750 : audit [DBG] from='client.? 172.21.15.139:0/1999663696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:51.105 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:51.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.248+0000 7fef7a184700 1 -- 172.21.15.139:0/3872634929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef740f6840 msgr2=0x7fef740f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:51.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.248+0000 7fef7a184700 1 --2- 172.21.15.139:0/3872634929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef740f6840 0x7fef740f6ca0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fef68004660 tx=0x7fef68009b30).stop 2021-11-20T06:06:51.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.248+0000 7fef7a184700 1 -- 172.21.15.139:0/3872634929 shutdown_connections 2021-11-20T06:06:51.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.248+0000 7fef7a184700 1 --2- 172.21.15.139:0/3872634929 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef740f6840 0x7fef740f6ca0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:51.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.248+0000 7fef7a184700 1 --2- 172.21.15.139:0/3872634929 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef740f55a0 0x7fef740f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:51.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.248+0000 7fef7a184700 1 -- 172.21.15.139:0/3872634929 >> 172.21.15.139:0/3872634929 conn(0x7fef740f0ce0 msgr2=0x7fef740f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:51.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.248+0000 7fef7a184700 1 -- 172.21.15.139:0/3872634929 shutdown_connections 2021-11-20T06:06:51.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.248+0000 7fef7a184700 1 -- 172.21.15.139:0/3872634929 wait complete. 2021-11-20T06:06:51.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.248+0000 7fef7a184700 1 Processor -- start 2021-11-20T06:06:51.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.249+0000 7fef7a184700 1 -- start start 2021-11-20T06:06:51.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.249+0000 7fef7a184700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef740f55a0 0x7fef74105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:51.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.249+0000 7fef7a184700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef740f6840 0x7fef74106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:51.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.249+0000 7fef7a184700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef74106840 con 0x7fef740f6840 2021-11-20T06:06:51.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.249+0000 7fef7a184700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef74106980 con 0x7fef740f55a0 2021-11-20T06:06:51.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.249+0000 7fef72ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef740f6840 0x7fef74106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:51.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.249+0000 7fef72ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef740f6840 0x7fef74106290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58484/0 (socket says 172.21.15.139:58484) 2021-11-20T06:06:51.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.249+0000 7fef72ffd700 1 -- 172.21.15.139:0/2059707586 learned_addr learned my addr 172.21.15.139:0/2059707586 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:51.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.249+0000 7fef737fe700 1 --2- 172.21.15.139:0/2059707586 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef740f55a0 0x7fef74105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:51.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.249+0000 7fef72ffd700 1 -- 172.21.15.139:0/2059707586 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef740f55a0 msgr2=0x7fef74105da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:51.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.249+0000 7fef72ffd700 1 --2- 172.21.15.139:0/2059707586 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef740f55a0 0x7fef74105da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:51.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.249+0000 7fef72ffd700 1 -- 172.21.15.139:0/2059707586 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef68005040 con 0x7fef740f6840 2021-11-20T06:06:51.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.249+0000 7fef72ffd700 1 --2- 172.21.15.139:0/2059707586 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef740f6840 0x7fef74106290 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fef68009f50 tx=0x7fef68009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:51.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.249+0000 7fef5bfff700 1 -- 172.21.15.139:0/2059707586 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef68018070 con 0x7fef740f6840 2021-11-20T06:06:51.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.250+0000 7fef7a184700 1 -- 172.21.15.139:0/2059707586 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7410b3c0 con 0x7fef740f6840 2021-11-20T06:06:51.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.250+0000 7fef5bfff700 1 -- 172.21.15.139:0/2059707586 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef68013410 con 0x7fef740f6840 2021-11-20T06:06:51.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.250+0000 7fef5bfff700 1 -- 172.21.15.139:0/2059707586 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef6801c3c0 con 0x7fef740f6840 2021-11-20T06:06:51.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.250+0000 7fef7a184700 1 -- 172.21.15.139:0/2059707586 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef7410b980 con 0x7fef740f6840 2021-11-20T06:06:51.246 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.250+0000 7fef7a184700 1 -- 172.21.15.139:0/2059707586 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef74027a00 con 0x7fef740f6840 2021-11-20T06:06:51.246 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.251+0000 7fef5bfff700 1 -- 172.21.15.139:0/2059707586 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fef6801d420 con 0x7fef740f6840 2021-11-20T06:06:51.246 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.251+0000 7fef5bfff700 1 --2- 172.21.15.139:0/2059707586 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fef5c05c2a0 0x7fef5c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:51.246 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.251+0000 7fef5bfff700 1 -- 172.21.15.139:0/2059707586 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fef68077ef0 con 0x7fef740f6840 2021-11-20T06:06:51.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.251+0000 7fef737fe700 1 --2- 172.21.15.139:0/2059707586 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fef5c05c2a0 0x7fef5c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:51.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.252+0000 7fef737fe700 1 --2- 172.21.15.139:0/2059707586 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fef5c05c2a0 0x7fef5c05e760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fef740f66a0 tx=0x7fef6400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:51.249 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.255+0000 7fef5bfff700 1 -- 172.21.15.139:0/2059707586 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef6804a800 con 0x7fef740f6840 2021-11-20T06:06:51.423 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.438+0000 7fef7a184700 1 -- 172.21.15.139:0/2059707586 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef740fdd30 con 0x7fef740f6840 2021-11-20T06:06:51.423 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.438+0000 7fef5bfff700 1 -- 172.21.15.139:0/2059707586 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2562 (secure 0 0 0) 0x7fef6804de20 con 0x7fef740f6840 2021-11-20T06:06:51.424 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:51.424 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":64,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:44.233143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14660":{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:51.427 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.440+0000 7fef7a184700 1 -- 172.21.15.139:0/2059707586 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fef5c05c2a0 msgr2=0x7fef5c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:51.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.440+0000 7fef7a184700 1 --2- 172.21.15.139:0/2059707586 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fef5c05c2a0 0x7fef5c05e760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fef740f66a0 tx=0x7fef6400b040).stop 2021-11-20T06:06:51.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.440+0000 7fef7a184700 1 -- 172.21.15.139:0/2059707586 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef740f6840 msgr2=0x7fef74106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:51.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.440+0000 7fef7a184700 1 --2- 172.21.15.139:0/2059707586 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef740f6840 0x7fef74106290 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fef68009f50 tx=0x7fef68009b30).stop 2021-11-20T06:06:51.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.440+0000 7fef7a184700 1 -- 172.21.15.139:0/2059707586 shutdown_connections 2021-11-20T06:06:51.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.440+0000 7fef7a184700 1 --2- 172.21.15.139:0/2059707586 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fef5c05c2a0 0x7fef5c05e760 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:51.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.440+0000 7fef7a184700 1 --2- 172.21.15.139:0/2059707586 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef740f55a0 0x7fef74105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:51.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.440+0000 7fef7a184700 1 --2- 172.21.15.139:0/2059707586 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef740f6840 0x7fef74106290 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:51.430 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.440+0000 7fef7a184700 1 -- 172.21.15.139:0/2059707586 >> 172.21.15.139:0/2059707586 conn(0x7fef740f0ce0 msgr2=0x7fef740fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:51.430 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.440+0000 7fef7a184700 1 -- 172.21.15.139:0/2059707586 shutdown_connections 2021-11-20T06:06:51.430 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:51.440+0000 7fef7a184700 1 -- 172.21.15.139:0/2059707586 wait complete. 2021-11-20T06:06:51.431 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 64 2021-11-20T06:06:51.437 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:44.233143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14660': {'gid': 14660, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6827/811251312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 811251312}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 811251312}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:51.437 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14660:up:rejoin 2021-11-20T06:06:51.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:51 smithi139 conmon[33501]: cluster 2021-11-20T06:06:50.262697+0000 2021-11-20T06:06:51.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:51 smithi139 conmon[33501]: mgr.smithi139.dplyxn (mgr.14162) 291 : cluster [DBG] pgmap v263: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:51.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:51 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:06:51.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:51 smithi139 conmon[33501]: 20T06:06:51.439189+0000 mon.smithi139 (mon.0) 751 : audit [DBG] from='client.? 172.21.15.139:0/2059707586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:51.941 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:51 smithi178 conmon[36600]: cluster 2021-11-20T06:06:50.262697+0000 mgr.smithi139.dplyxn (mgr.14162) 291 : cluster [DBG] pgmap v263: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:51.941 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:51 smithi178 conmon[36600]: audit 2021-11-20T06:06:51.439189+0000 mon.smithi139 (mon.0) 751 : audit [DBG] from='client.? 172.21.15.139:0/2059707586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:52.439 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:52.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.594+0000 7f3e0aa34700 1 -- 172.21.15.139:0/3345254476 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e040f55b0 msgr2=0x7f3e040f59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:52.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.594+0000 7f3e0aa34700 1 --2- 172.21.15.139:0/3345254476 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e040f55b0 0x7f3e040f59d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f3df4004660 tx=0x7f3df4009b30).stop 2021-11-20T06:06:52.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.594+0000 7f3e0aa34700 1 -- 172.21.15.139:0/3345254476 shutdown_connections 2021-11-20T06:06:52.582 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.594+0000 7f3e0aa34700 1 --2- 172.21.15.139:0/3345254476 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e040f6850 0x7f3e040f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:52.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.594+0000 7f3e0aa34700 1 --2- 172.21.15.139:0/3345254476 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e040f55b0 0x7f3e040f59d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:52.584 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.594+0000 7f3e0aa34700 1 -- 172.21.15.139:0/3345254476 >> 172.21.15.139:0/3345254476 conn(0x7f3e040f0d10 msgr2=0x7f3e040f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:52.584 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.594+0000 7f3e0aa34700 1 -- 172.21.15.139:0/3345254476 shutdown_connections 2021-11-20T06:06:52.586 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.594+0000 7f3e0aa34700 1 -- 172.21.15.139:0/3345254476 wait complete. 2021-11-20T06:06:52.587 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.594+0000 7f3e0aa34700 1 Processor -- start 2021-11-20T06:06:52.592 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.594+0000 7f3e0aa34700 1 -- start start 2021-11-20T06:06:52.593 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e0aa34700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e040f55b0 0x7f3e04060530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:52.593 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e0aa34700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e040f6850 0x7f3e0410e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:52.594 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e0aa34700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e04107a80 con 0x7f3e040f6850 2021-11-20T06:06:52.594 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e0aa34700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0410e590 con 0x7f3e040f55b0 2021-11-20T06:06:52.605 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e037fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e040f6850 0x7f3e0410e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:52.607 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e037fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e040f6850 0x7f3e0410e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58494/0 (socket says 172.21.15.139:58494) 2021-11-20T06:06:52.608 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e037fe700 1 -- 172.21.15.139:0/3401361859 learned_addr learned my addr 172.21.15.139:0/3401361859 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:52.608 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e03fff700 1 --2- 172.21.15.139:0/3401361859 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e040f55b0 0x7f3e04060530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:52.609 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e037fe700 1 -- 172.21.15.139:0/3401361859 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e040f55b0 msgr2=0x7f3e04060530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:52.611 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e037fe700 1 --2- 172.21.15.139:0/3401361859 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e040f55b0 0x7f3e04060530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:52.611 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e037fe700 1 -- 172.21.15.139:0/3401361859 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df4005040 con 0x7f3e040f6850 2021-11-20T06:06:52.612 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e037fe700 1 --2- 172.21.15.139:0/3401361859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e040f6850 0x7f3e0410e0a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f3df800c560 tx=0x7f3df8007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:52.612 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e00ff9700 1 -- 172.21.15.139:0/3401361859 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3df800f650 con 0x7f3e040f6850 2021-11-20T06:06:52.613 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e00ff9700 1 -- 172.21.15.139:0/3401361859 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3df800fc60 con 0x7f3e040f6850 2021-11-20T06:06:52.614 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e0aa34700 1 -- 172.21.15.139:0/3401361859 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0410e730 con 0x7f3e040f6850 2021-11-20T06:06:52.614 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e00ff9700 1 -- 172.21.15.139:0/3401361859 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3df8012c80 con 0x7f3e040f6850 2021-11-20T06:06:52.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.595+0000 7f3e0aa34700 1 -- 172.21.15.139:0/3401361859 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0410ecf0 con 0x7f3e040f6850 2021-11-20T06:06:52.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.596+0000 7f3e0aa34700 1 -- 172.21.15.139:0/3401361859 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e04027a00 con 0x7f3e040f6850 2021-11-20T06:06:52.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.597+0000 7f3e00ff9700 1 -- 172.21.15.139:0/3401361859 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3df8010070 con 0x7f3e040f6850 2021-11-20T06:06:52.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.597+0000 7f3e00ff9700 1 --2- 172.21.15.139:0/3401361859 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3dec05c2a0 0x7f3dec05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:52.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.597+0000 7f3e00ff9700 1 -- 172.21.15.139:0/3401361859 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3df8075fb0 con 0x7f3e040f6850 2021-11-20T06:06:52.617 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.597+0000 7f3e03fff700 1 --2- 172.21.15.139:0/3401361859 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3dec05c2a0 0x7f3dec05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:52.617 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.598+0000 7f3e03fff700 1 --2- 172.21.15.139:0/3401361859 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3dec05c2a0 0x7f3dec05e760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f3df4009d60 tx=0x7f3df4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:52.621 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.601+0000 7f3e00ff9700 1 -- 172.21.15.139:0/3401361859 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3df80452b0 con 0x7f3e040f6850 2021-11-20T06:06:52.766 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.782+0000 7f3e0aa34700 1 -- 172.21.15.139:0/3401361859 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e040fdd40 con 0x7f3e040f6850 2021-11-20T06:06:52.767 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.782+0000 7f3e00ff9700 1 -- 172.21.15.139:0/3401361859 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2562 (secure 0 0 0) 0x7f3df804c2f0 con 0x7f3e040f6850 2021-11-20T06:06:52.768 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:52.768 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":64,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:44.233143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14660":{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:52.769 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.784+0000 7f3e0aa34700 1 -- 172.21.15.139:0/3401361859 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3dec05c2a0 msgr2=0x7f3dec05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:52.770 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.784+0000 7f3e0aa34700 1 --2- 172.21.15.139:0/3401361859 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3dec05c2a0 0x7f3dec05e760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f3df4009d60 tx=0x7f3df4009b30).stop 2021-11-20T06:06:52.770 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.784+0000 7f3e0aa34700 1 -- 172.21.15.139:0/3401361859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e040f6850 msgr2=0x7f3e0410e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:52.770 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.784+0000 7f3e0aa34700 1 --2- 172.21.15.139:0/3401361859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e040f6850 0x7f3e0410e0a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f3df800c560 tx=0x7f3df8007e20).stop 2021-11-20T06:06:52.771 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.784+0000 7f3e0aa34700 1 -- 172.21.15.139:0/3401361859 shutdown_connections 2021-11-20T06:06:52.771 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.784+0000 7f3e0aa34700 1 --2- 172.21.15.139:0/3401361859 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3dec05c2a0 0x7f3dec05e760 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:52.771 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.784+0000 7f3e0aa34700 1 --2- 172.21.15.139:0/3401361859 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e040f55b0 0x7f3e04060530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:52.771 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.784+0000 7f3e0aa34700 1 --2- 172.21.15.139:0/3401361859 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3e040f6850 0x7f3e0410e0a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:52.771 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.784+0000 7f3e0aa34700 1 -- 172.21.15.139:0/3401361859 >> 172.21.15.139:0/3401361859 conn(0x7f3e040f0d10 msgr2=0x7f3e040fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:52.772 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.784+0000 7f3e0aa34700 1 -- 172.21.15.139:0/3401361859 shutdown_connections 2021-11-20T06:06:52.772 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:52.784+0000 7f3e0aa34700 1 -- 172.21.15.139:0/3401361859 wait complete. 2021-11-20T06:06:52.773 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 64 2021-11-20T06:06:52.780 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:44.233143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14660': {'gid': 14660, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6827/811251312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 811251312}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 811251312}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:52.781 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14660:up:rejoin 2021-11-20T06:06:53.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:53 smithi139 conmon[33501]: cluster 2021-11-20T06:06:52.263127+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:06:53.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:53 smithi139 conmon[33501]: 14162) 292 : cluster [DBG] pgmap v264: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:53.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:53 smithi139 conmon[33501]: audit 2021-11-20T06:06:52.782915+0000 mon.smithi139 (mon.0) 752 : audit [DBG] from='client.? 172.21.15.139:0/3401361859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:53.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:53 smithi178 conmon[36600]: cluster 2021-11-20T06:06:52.263127+0000 mgr.smithi139.dplyxn (mgr.14162) 292 : cluster 2021-11-20T06:06:53.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:53 smithi178 conmon[36600]: [DBG] pgmap v264: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:06:53.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:53 smithi178 conmon[36600]: audit 2021-11-20T06:06:52.782915+0000 mon.smithi139 (mon.0) 752 : audit [DBG] from='client.? 172.21.15.139:0/3401361859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:53.782 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:53.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.922+0000 7f39534bc700 1 -- 172.21.15.139:0/1646902570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f394c0f9800 msgr2=0x7f394c0fbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:53.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.922+0000 7f39534bc700 1 --2- 172.21.15.139:0/1646902570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f394c0f9800 0x7f394c0fbc50 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f3940004660 tx=0x7f3940009b30).stop 2021-11-20T06:06:53.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.922+0000 7f39534bc700 1 -- 172.21.15.139:0/1646902570 shutdown_connections 2021-11-20T06:06:53.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.922+0000 7f39534bc700 1 --2- 172.21.15.139:0/1646902570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f394c0f9800 0x7f394c0fbc50 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:53.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.922+0000 7f39534bc700 1 --2- 172.21.15.139:0/1646902570 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f394c060140 0x7f394c0f9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:53.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.922+0000 7f39534bc700 1 -- 172.21.15.139:0/1646902570 >> 172.21.15.139:0/1646902570 conn(0x7f394c0f0ce0 msgr2=0x7f394c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:53.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.922+0000 7f39534bc700 1 -- 172.21.15.139:0/1646902570 shutdown_connections 2021-11-20T06:06:53.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.922+0000 7f39534bc700 1 -- 172.21.15.139:0/1646902570 wait complete. 2021-11-20T06:06:53.911 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.922+0000 7f39534bc700 1 Processor -- start 2021-11-20T06:06:53.911 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.922+0000 7f39534bc700 1 -- start start 2021-11-20T06:06:53.911 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f39534bc700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f394c060140 0x7f394c106e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:53.911 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f39534bc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f394c0f9800 0x7f394c107360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:53.912 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f39534bc700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c107910 con 0x7f394c0f9800 2021-11-20T06:06:53.912 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f39534bc700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c107a50 con 0x7f394c060140 2021-11-20T06:06:53.912 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f3950a57700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f394c0f9800 0x7f394c107360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:53.912 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f3950a57700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f394c0f9800 0x7f394c107360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58506/0 (socket says 172.21.15.139:58506) 2021-11-20T06:06:53.912 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f3950a57700 1 -- 172.21.15.139:0/3783570478 learned_addr learned my addr 172.21.15.139:0/3783570478 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:53.913 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f3951258700 1 --2- 172.21.15.139:0/3783570478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f394c060140 0x7f394c106e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:53.913 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f3950a57700 1 -- 172.21.15.139:0/3783570478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f394c060140 msgr2=0x7f394c106e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:53.913 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f3950a57700 1 --2- 172.21.15.139:0/3783570478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f394c060140 0x7f394c106e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:53.913 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f3950a57700 1 -- 172.21.15.139:0/3783570478 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3940005040 con 0x7f394c0f9800 2021-11-20T06:06:53.914 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f3951258700 1 --2- 172.21.15.139:0/3783570478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f394c060140 0x7f394c106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:06:53.914 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f3950a57700 1 --2- 172.21.15.139:0/3783570478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f394c0f9800 0x7f394c107360 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f3940009f50 tx=0x7f3940009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:53.914 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f3949ffb700 1 -- 172.21.15.139:0/3783570478 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3940018070 con 0x7f394c0f9800 2021-11-20T06:06:53.914 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f3949ffb700 1 -- 172.21.15.139:0/3783570478 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3940013410 con 0x7f394c0f9800 2021-11-20T06:06:53.914 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f39534bc700 1 -- 172.21.15.139:0/3783570478 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394c10c490 con 0x7f394c0f9800 2021-11-20T06:06:53.915 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.923+0000 7f3949ffb700 1 -- 172.21.15.139:0/3783570478 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f394001c620 con 0x7f394c0f9800 2021-11-20T06:06:53.915 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.924+0000 7f39534bc700 1 -- 172.21.15.139:0/3783570478 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f394c10ca50 con 0x7f394c0f9800 2021-11-20T06:06:53.915 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.924+0000 7f39534bc700 1 -- 172.21.15.139:0/3783570478 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f394c027a00 con 0x7f394c0f9800 2021-11-20T06:06:53.916 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.925+0000 7f3949ffb700 1 -- 172.21.15.139:0/3783570478 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f394001d420 con 0x7f394c0f9800 2021-11-20T06:06:53.917 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.925+0000 7f3949ffb700 1 --2- 172.21.15.139:0/3783570478 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f393405c2a0 0x7f393405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:53.917 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.925+0000 7f3949ffb700 1 -- 172.21.15.139:0/3783570478 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3940078060 con 0x7f394c0f9800 2021-11-20T06:06:53.917 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.925+0000 7f3951258700 1 --2- 172.21.15.139:0/3783570478 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f393405c2a0 0x7f393405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:53.917 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.926+0000 7f3951258700 1 --2- 172.21.15.139:0/3783570478 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f393405c2a0 0x7f393405e760 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f393c004550 tx=0x7f393c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:53.918 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:53.929+0000 7f3949ffb700 1 -- 172.21.15.139:0/3783570478 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f394004a9f0 con 0x7f394c0f9800 2021-11-20T06:06:54.098 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:54.114+0000 7f39534bc700 1 -- 172.21.15.139:0/3783570478 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f394c0fee30 con 0x7f394c0f9800 2021-11-20T06:06:54.099 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:54.114+0000 7f3949ffb700 1 -- 172.21.15.139:0/3783570478 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2562 (secure 0 0 0) 0x7f394004e010 con 0x7f394c0f9800 2021-11-20T06:06:54.099 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:54.100 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":64,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:44.233143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14660":{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:54.101 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:54.116+0000 7f39534bc700 1 -- 172.21.15.139:0/3783570478 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f393405c2a0 msgr2=0x7f393405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:54.101 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:54.116+0000 7f39534bc700 1 --2- 172.21.15.139:0/3783570478 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f393405c2a0 0x7f393405e760 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f393c004550 tx=0x7f393c009040).stop 2021-11-20T06:06:54.102 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:54.116+0000 7f39534bc700 1 -- 172.21.15.139:0/3783570478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f394c0f9800 msgr2=0x7f394c107360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:54.102 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:54.116+0000 7f39534bc700 1 --2- 172.21.15.139:0/3783570478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f394c0f9800 0x7f394c107360 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f3940009f50 tx=0x7f3940009b30).stop 2021-11-20T06:06:54.102 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:54.116+0000 7f39534bc700 1 -- 172.21.15.139:0/3783570478 shutdown_connections 2021-11-20T06:06:54.102 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:54.116+0000 7f39534bc700 1 --2- 172.21.15.139:0/3783570478 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f393405c2a0 0x7f393405e760 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:54.103 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:54.116+0000 7f39534bc700 1 --2- 172.21.15.139:0/3783570478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f394c060140 0x7f394c106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:54.103 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:54.116+0000 7f39534bc700 1 --2- 172.21.15.139:0/3783570478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f394c0f9800 0x7f394c107360 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:54.103 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:54.116+0000 7f39534bc700 1 -- 172.21.15.139:0/3783570478 >> 172.21.15.139:0/3783570478 conn(0x7f394c0f0ce0 msgr2=0x7f394c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:54.103 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:54.116+0000 7f39534bc700 1 -- 172.21.15.139:0/3783570478 shutdown_connections 2021-11-20T06:06:54.103 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:54.116+0000 7f39534bc700 1 -- 172.21.15.139:0/3783570478 wait complete. 2021-11-20T06:06:54.104 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 64 2021-11-20T06:06:54.113 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:44.233143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14660': {'gid': 14660, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6827/811251312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 811251312}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 811251312}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:54.113 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14660:up:rejoin 2021-11-20T06:06:54.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:54 smithi139 conmon[33501]: audit 2021-11-20T06:06:54.115233+0000 mon.smithi139 (mon 2021-11-20T06:06:54.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:54 smithi139 conmon[33501]: .0) 753 : audit [DBG] from='client.? 172.21.15.139:0/3783570478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:54.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:54 smithi178 conmon[36600]: audit 2021-11-20T06:06:54.115233+0000 mon.smithi139 (mon.0) 753 : audit [DBG] from='client.? 172.21.15.139:0/3783570478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:55.114 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:55.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.260+0000 7f6dd95a5700 1 -- 172.21.15.139:0/2629487284 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6dd40f5300 msgr2=0x7f6dd40f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:55.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.260+0000 7f6dd95a5700 1 --2- 172.21.15.139:0/2629487284 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6dd40f5300 0x7f6dd40f7a80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f6dc8004660 tx=0x7f6dc8009b30).stop 2021-11-20T06:06:55.248 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.260+0000 7f6dd95a5700 1 -- 172.21.15.139:0/2629487284 shutdown_connections 2021-11-20T06:06:55.248 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.260+0000 7f6dd95a5700 1 --2- 172.21.15.139:0/2629487284 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6dd40f5300 0x7f6dd40f7a80 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:55.248 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.260+0000 7f6dd95a5700 1 --2- 172.21.15.139:0/2629487284 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6dd40f4960 0x7f6dd40f4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:55.248 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.260+0000 7f6dd95a5700 1 -- 172.21.15.139:0/2629487284 >> 172.21.15.139:0/2629487284 conn(0x7f6dd40f0440 msgr2=0x7f6dd40f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:55.249 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.260+0000 7f6dd95a5700 1 -- 172.21.15.139:0/2629487284 shutdown_connections 2021-11-20T06:06:55.249 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.260+0000 7f6dd95a5700 1 -- 172.21.15.139:0/2629487284 wait complete. 2021-11-20T06:06:55.249 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd95a5700 1 Processor -- start 2021-11-20T06:06:55.250 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd95a5700 1 -- start start 2021-11-20T06:06:55.251 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd95a5700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6dd40f4960 0x7f6dd4106f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:55.252 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd95a5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6dd40f5300 0x7f6dd4107410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:55.252 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd95a5700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd4107a50 con 0x7f6dd40f5300 2021-11-20T06:06:55.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd95a5700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd4107b90 con 0x7f6dd40f4960 2021-11-20T06:06:55.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd27fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6dd40f5300 0x7f6dd4107410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:55.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd27fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6dd40f5300 0x7f6dd4107410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58514/0 (socket says 172.21.15.139:58514) 2021-11-20T06:06:55.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd27fc700 1 -- 172.21.15.139:0/2411418624 learned_addr learned my addr 172.21.15.139:0/2411418624 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:55.254 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd2ffd700 1 --2- 172.21.15.139:0/2411418624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6dd40f4960 0x7f6dd4106f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:55.254 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd27fc700 1 -- 172.21.15.139:0/2411418624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6dd40f4960 msgr2=0x7f6dd4106f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:55.254 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd27fc700 1 --2- 172.21.15.139:0/2411418624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6dd40f4960 0x7f6dd4106f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:55.254 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd27fc700 1 -- 172.21.15.139:0/2411418624 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc8005040 con 0x7f6dd40f5300 2021-11-20T06:06:55.254 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd2ffd700 1 --2- 172.21.15.139:0/2411418624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6dd40f4960 0x7f6dd4106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:06:55.255 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd27fc700 1 --2- 172.21.15.139:0/2411418624 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6dd40f5300 0x7f6dd4107410 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f6dc8009f50 tx=0x7f6dc8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:55.255 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dbb7fe700 1 -- 172.21.15.139:0/2411418624 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dc8018070 con 0x7f6dd40f5300 2021-11-20T06:06:55.255 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dbb7fe700 1 -- 172.21.15.139:0/2411418624 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6dc8013410 con 0x7f6dd40f5300 2021-11-20T06:06:55.255 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd95a5700 1 -- 172.21.15.139:0/2411418624 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd410c5d0 con 0x7f6dd40f5300 2021-11-20T06:06:55.256 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dbb7fe700 1 -- 172.21.15.139:0/2411418624 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dc801c620 con 0x7f6dd40f5300 2021-11-20T06:06:55.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.261+0000 7f6dd95a5700 1 -- 172.21.15.139:0/2411418624 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd410cb30 con 0x7f6dd40f5300 2021-11-20T06:06:55.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.262+0000 7f6dd95a5700 1 -- 172.21.15.139:0/2411418624 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd4027a00 con 0x7f6dd40f5300 2021-11-20T06:06:55.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.263+0000 7f6dbb7fe700 1 -- 172.21.15.139:0/2411418624 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6dc801d420 con 0x7f6dd40f5300 2021-11-20T06:06:55.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.263+0000 7f6dbb7fe700 1 --2- 172.21.15.139:0/2411418624 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6dbc05c2a0 0x7f6dbc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:55.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.263+0000 7f6dbb7fe700 1 -- 172.21.15.139:0/2411418624 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6dc8078120 con 0x7f6dd40f5300 2021-11-20T06:06:55.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.263+0000 7f6dd2ffd700 1 --2- 172.21.15.139:0/2411418624 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6dbc05c2a0 0x7f6dbc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:55.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.263+0000 7f6dd2ffd700 1 --2- 172.21.15.139:0/2411418624 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6dbc05c2a0 0x7f6dbc05e760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f6dd405f7e0 tx=0x7f6dc4008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:55.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.267+0000 7f6dbb7fe700 1 -- 172.21.15.139:0/2411418624 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6dc804a9d0 con 0x7f6dd40f5300 2021-11-20T06:06:55.436 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.451+0000 7f6dd95a5700 1 -- 172.21.15.139:0/2411418624 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6dd40feef0 con 0x7f6dd40f5300 2021-11-20T06:06:55.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.452+0000 7f6dbb7fe700 1 -- 172.21.15.139:0/2411418624 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2562 (secure 0 0 0) 0x7f6dc804dff0 con 0x7f6dd40f5300 2021-11-20T06:06:55.437 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:55.437 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":64,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:44.233143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14660":{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:55.439 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.453+0000 7f6dd95a5700 1 -- 172.21.15.139:0/2411418624 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6dbc05c2a0 msgr2=0x7f6dbc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:55.439 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.453+0000 7f6dd95a5700 1 --2- 172.21.15.139:0/2411418624 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6dbc05c2a0 0x7f6dbc05e760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f6dd405f7e0 tx=0x7f6dc4008040).stop 2021-11-20T06:06:55.440 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.453+0000 7f6dd95a5700 1 -- 172.21.15.139:0/2411418624 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6dd40f5300 msgr2=0x7f6dd4107410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:55.441 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.453+0000 7f6dd95a5700 1 --2- 172.21.15.139:0/2411418624 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6dd40f5300 0x7f6dd4107410 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f6dc8009f50 tx=0x7f6dc8009b30).stop 2021-11-20T06:06:55.441 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.453+0000 7f6dd95a5700 1 -- 172.21.15.139:0/2411418624 shutdown_connections 2021-11-20T06:06:55.441 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.453+0000 7f6dd95a5700 1 --2- 172.21.15.139:0/2411418624 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6dbc05c2a0 0x7f6dbc05e760 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:55.442 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.453+0000 7f6dd95a5700 1 --2- 172.21.15.139:0/2411418624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6dd40f4960 0x7f6dd4106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:55.443 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.453+0000 7f6dd95a5700 1 --2- 172.21.15.139:0/2411418624 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6dd40f5300 0x7f6dd4107410 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:55.443 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.453+0000 7f6dd95a5700 1 -- 172.21.15.139:0/2411418624 >> 172.21.15.139:0/2411418624 conn(0x7f6dd40f0440 msgr2=0x7f6dd40fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:55.443 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.454+0000 7f6dd95a5700 1 -- 172.21.15.139:0/2411418624 shutdown_connections 2021-11-20T06:06:55.443 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:55.454+0000 7f6dd95a5700 1 -- 172.21.15.139:0/2411418624 wait complete. 2021-11-20T06:06:55.444 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 64 2021-11-20T06:06:55.449 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:55 smithi139 conmon[33501]: cluster 2021-11-20T06:06:54.263586+0000 mgr.smithi139.dplyxn (mgr.14162) 293 : cluster [DBG] pgmap v265: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:06:55.450 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:44.233143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14660': {'gid': 14660, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6827/811251312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 811251312}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 811251312}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:55.451 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14660:up:rejoin 2021-11-20T06:06:55.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:55 smithi178 conmon[36600]: cluster 2021-11-20T06:06:54.263586+0000 mgr.smithi139.dplyxn (mgr.14162) 293 : cluster 2021-11-20T06:06:55.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:55 smithi178 conmon[36600]: [DBG] pgmap v265: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:06:56.452 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:56.480 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:56 smithi139 conmon[33501]: audit 2021-11-20T06:06:55.452631+0000 mon.smithi139 (mon.0) 754 : audit [DBG] from='client.? 172.21.15.139:0/2411418624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:56.480 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:56 smithi139 conmon[33501]: 2021-11-20T06:06:56.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.593+0000 7fc2d2060700 1 -- 172.21.15.139:0/361339156 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2cc0603d0 msgr2=0x7fc2cc0f92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:56.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.593+0000 7fc2d2060700 1 --2- 172.21.15.139:0/361339156 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2cc0603d0 0x7fc2cc0f92a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fc2c40099e0 tx=0x7fc2c4004d10).stop 2021-11-20T06:06:56.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.593+0000 7fc2d2060700 1 -- 172.21.15.139:0/361339156 shutdown_connections 2021-11-20T06:06:56.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.593+0000 7fc2d2060700 1 --2- 172.21.15.139:0/361339156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2cc0f9790 0x7fc2cc0fbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:56.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.593+0000 7fc2d2060700 1 --2- 172.21.15.139:0/361339156 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2cc0603d0 0x7fc2cc0f92a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:56.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.593+0000 7fc2d2060700 1 -- 172.21.15.139:0/361339156 >> 172.21.15.139:0/361339156 conn(0x7fc2cc0f0c40 msgr2=0x7fc2cc0f3080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:56.582 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.593+0000 7fc2d2060700 1 -- 172.21.15.139:0/361339156 shutdown_connections 2021-11-20T06:06:56.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.593+0000 7fc2d2060700 1 -- 172.21.15.139:0/361339156 wait complete. 2021-11-20T06:06:56.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.594+0000 7fc2d2060700 1 Processor -- start 2021-11-20T06:06:56.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.594+0000 7fc2d2060700 1 -- start start 2021-11-20T06:06:56.584 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.594+0000 7fc2d2060700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2cc0603d0 0x7fc2cc0f6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:56.585 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.594+0000 7fc2d2060700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2cc0f9790 0x7fc2cc0f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:56.585 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.594+0000 7fc2d2060700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc0f5700 con 0x7fc2cc0f9790 2021-11-20T06:06:56.585 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.594+0000 7fc2d2060700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc0f5840 con 0x7fc2cc0603d0 2021-11-20T06:06:56.585 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.594+0000 7fc2d085d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2cc0f9790 0x7fc2cc0f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:56.586 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.594+0000 7fc2d085d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2cc0f9790 0x7fc2cc0f51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58524/0 (socket says 172.21.15.139:58524) 2021-11-20T06:06:56.586 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.594+0000 7fc2d085d700 1 -- 172.21.15.139:0/2366149279 learned_addr learned my addr 172.21.15.139:0/2366149279 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:56.586 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.594+0000 7fc2d085d700 1 -- 172.21.15.139:0/2366149279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2cc0603d0 msgr2=0x7fc2cc0f6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:56.586 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.594+0000 7fc2d085d700 1 --2- 172.21.15.139:0/2366149279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2cc0603d0 0x7fc2cc0f6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:56.587 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.594+0000 7fc2d085d700 1 -- 172.21.15.139:0/2366149279 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c4009640 con 0x7fc2cc0f9790 2021-11-20T06:06:56.587 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.595+0000 7fc2d085d700 1 --2- 172.21.15.139:0/2366149279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2cc0f9790 0x7fc2cc0f51a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fc2c00026d0 tx=0x7fc2c0009d40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:56.587 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.595+0000 7fc2c9ffb700 1 -- 172.21.15.139:0/2366149279 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2c0009320 con 0x7fc2cc0f9790 2021-11-20T06:06:56.587 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.595+0000 7fc2c9ffb700 1 -- 172.21.15.139:0/2366149279 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2c000f040 con 0x7fc2cc0f9790 2021-11-20T06:06:56.587 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.595+0000 7fc2c9ffb700 1 -- 172.21.15.139:0/2366149279 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2c0014710 con 0x7fc2cc0f9790 2021-11-20T06:06:56.589 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.595+0000 7fc2d2060700 1 -- 172.21.15.139:0/2366149279 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2cc0f5b20 con 0x7fc2cc0f9790 2021-11-20T06:06:56.589 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.595+0000 7fc2d2060700 1 -- 172.21.15.139:0/2366149279 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2cc10eeb0 con 0x7fc2cc0f9790 2021-11-20T06:06:56.591 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.596+0000 7fc2d2060700 1 -- 172.21.15.139:0/2366149279 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2cc045ad0 con 0x7fc2cc0f9790 2021-11-20T06:06:56.591 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.596+0000 7fc2c9ffb700 1 -- 172.21.15.139:0/2366149279 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc2c0009480 con 0x7fc2cc0f9790 2021-11-20T06:06:56.592 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.597+0000 7fc2c9ffb700 1 --2- 172.21.15.139:0/2366149279 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc2b405c300 0x7fc2b405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:56.592 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.597+0000 7fc2c9ffb700 1 -- 172.21.15.139:0/2366149279 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc2c00768f0 con 0x7fc2cc0f9790 2021-11-20T06:06:56.593 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.597+0000 7fc2d105e700 1 --2- 172.21.15.139:0/2366149279 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc2b405c300 0x7fc2b405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:56.593 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.597+0000 7fc2d105e700 1 --2- 172.21.15.139:0/2366149279 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc2b405c300 0x7fc2b405e7c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fc2c4000f80 tx=0x7fc2c4004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:56.594 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.600+0000 7fc2c9ffb700 1 -- 172.21.15.139:0/2366149279 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2c00491a0 con 0x7fc2cc0f9790 2021-11-20T06:06:56.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:56 smithi178 conmon[36600]: audit 2021-11-20T06:06:55.452631+0000 mon.smithi139 (mon.0) 754 2021-11-20T06:06:56.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:56 smithi178 conmon[36600]: : audit [DBG] from='client.? 172.21.15.139:0/2411418624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:56.769 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.785+0000 7fc2d2060700 1 -- 172.21.15.139:0/2366149279 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2cc027a00 con 0x7fc2cc0f9790 2021-11-20T06:06:56.770 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.786+0000 7fc2c9ffb700 1 -- 172.21.15.139:0/2366149279 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2562 (secure 0 0 0) 0x7fc2c004c7c0 con 0x7fc2cc0f9790 2021-11-20T06:06:56.771 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:56.771 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":64,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:44.233143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14660":{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:56.774 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.787+0000 7fc2d2060700 1 -- 172.21.15.139:0/2366149279 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc2b405c300 msgr2=0x7fc2b405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:56.774 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.787+0000 7fc2d2060700 1 --2- 172.21.15.139:0/2366149279 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc2b405c300 0x7fc2b405e7c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fc2c4000f80 tx=0x7fc2c4004d10).stop 2021-11-20T06:06:56.774 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.787+0000 7fc2d2060700 1 -- 172.21.15.139:0/2366149279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2cc0f9790 msgr2=0x7fc2cc0f51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:56.774 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.787+0000 7fc2d2060700 1 --2- 172.21.15.139:0/2366149279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2cc0f9790 0x7fc2cc0f51a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fc2c00026d0 tx=0x7fc2c0009d40).stop 2021-11-20T06:06:56.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.787+0000 7fc2d2060700 1 -- 172.21.15.139:0/2366149279 shutdown_connections 2021-11-20T06:06:56.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.787+0000 7fc2d2060700 1 --2- 172.21.15.139:0/2366149279 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc2b405c300 0x7fc2b405e7c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:56.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.787+0000 7fc2d2060700 1 --2- 172.21.15.139:0/2366149279 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2cc0603d0 0x7fc2cc0f6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:06:56.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.787+0000 7fc2d2060700 1 --2- 172.21.15.139:0/2366149279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2cc0f9790 0x7fc2cc0f51a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:56.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.787+0000 7fc2d2060700 1 -- 172.21.15.139:0/2366149279 >> 172.21.15.139:0/2366149279 conn(0x7fc2cc0f0c40 msgr2=0x7fc2cc0f3080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:56.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.787+0000 7fc2d2060700 1 -- 172.21.15.139:0/2366149279 shutdown_connections 2021-11-20T06:06:56.777 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:56.787+0000 7fc2d2060700 1 -- 172.21.15.139:0/2366149279 wait complete. 2021-11-20T06:06:56.778 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 64 2021-11-20T06:06:56.784 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:44.233143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14660': {'gid': 14660, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6827/811251312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 811251312}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 811251312}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:56.784 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14660:up:rejoin 2021-11-20T06:06:57.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:57 smithi139 conmon[33501]: cluster 2021-11-20T06:06:56.264065+0000 mgr.smithi139.dplyxn (mgr.14162) 294 2021-11-20T06:06:57.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:57 smithi139 conmon[33501]: : cluster [DBG] pgmap v266: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 0 op/s 2021-11-20T06:06:57.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:57 smithi139 conmon[33501]: audit 2021-11-20T06:06:56.786334+0000 mon.smithi139 (mon.0) 755 : audit [DBG] from='client.? 172.21.15.139:0/2366149279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:57.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:57 smithi178 conmon[36600]: cluster 2021-11-20T06:06:56.264065+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:06:57.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:57 smithi178 conmon[36600]: 14162) 294 : cluster [DBG] pgmap v266: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 0 op/s 2021-11-20T06:06:57.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:57 smithi178 conmon[36600]: audit 2021-11-20T06:06:56.786334+0000 mon.smithi139 (mon.0) 755 : audit [DBG] from='client.? 172.21.15.139:0/2366149279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:57.786 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:57.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.933+0000 7f5895ef3700 1 -- 172.21.15.139:0/1373514453 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58900f55a0 msgr2=0x7f58900f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:58.000 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.933+0000 7f5895ef3700 1 --2- 172.21.15.139:0/1373514453 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58900f55a0 0x7f58900f59c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f5880004660 tx=0x7f5880009b30).stop 2021-11-20T06:06:58.000 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.933+0000 7f5895ef3700 1 -- 172.21.15.139:0/1373514453 shutdown_connections 2021-11-20T06:06:58.000 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.933+0000 7f5895ef3700 1 --2- 172.21.15.139:0/1373514453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58900f6840 0x7f58900f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:58.001 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.933+0000 7f5895ef3700 1 --2- 172.21.15.139:0/1373514453 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58900f55a0 0x7f58900f59c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:58.001 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.933+0000 7f5895ef3700 1 -- 172.21.15.139:0/1373514453 >> 172.21.15.139:0/1373514453 conn(0x7f58900f0ce0 msgr2=0x7f58900f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:58.001 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.933+0000 7f5895ef3700 1 -- 172.21.15.139:0/1373514453 shutdown_connections 2021-11-20T06:06:58.001 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.933+0000 7f5895ef3700 1 -- 172.21.15.139:0/1373514453 wait complete. 2021-11-20T06:06:58.002 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.933+0000 7f5895ef3700 1 Processor -- start 2021-11-20T06:06:58.002 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f5895ef3700 1 -- start start 2021-11-20T06:06:58.002 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f5895ef3700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58900f55a0 0x7f5890106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:58.002 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f5895ef3700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58900f6840 0x7f58901073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:58.002 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f5895ef3700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5890107950 con 0x7f58900f55a0 2021-11-20T06:06:58.003 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f5895ef3700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5890107a90 con 0x7f58900f6840 2021-11-20T06:06:58.003 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f588f7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58900f55a0 0x7f5890106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:58.003 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f588f7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58900f55a0 0x7f5890106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58532/0 (socket says 172.21.15.139:58532) 2021-11-20T06:06:58.003 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f588f7fe700 1 -- 172.21.15.139:0/2191664297 learned_addr learned my addr 172.21.15.139:0/2191664297 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:58.004 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f588effd700 1 --2- 172.21.15.139:0/2191664297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58900f6840 0x7f58901073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:58.004 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f588f7fe700 1 -- 172.21.15.139:0/2191664297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58900f6840 msgr2=0x7f58901073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:58.004 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f588f7fe700 1 --2- 172.21.15.139:0/2191664297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58900f6840 0x7f58901073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:58.004 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f588f7fe700 1 -- 172.21.15.139:0/2191664297 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5880005040 con 0x7f58900f55a0 2021-11-20T06:06:58.004 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f588f7fe700 1 --2- 172.21.15.139:0/2191664297 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58900f55a0 0x7f5890106eb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f5880009f50 tx=0x7f5880009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:58.005 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f5877fff700 1 -- 172.21.15.139:0/2191664297 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5880018070 con 0x7f58900f55a0 2021-11-20T06:06:58.005 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f5895ef3700 1 -- 172.21.15.139:0/2191664297 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f589010c4d0 con 0x7f58900f55a0 2021-11-20T06:06:58.005 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.934+0000 7f5877fff700 1 -- 172.21.15.139:0/2191664297 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5880013410 con 0x7f58900f55a0 2021-11-20T06:06:58.005 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.935+0000 7f5877fff700 1 -- 172.21.15.139:0/2191664297 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f588001c620 con 0x7f58900f55a0 2021-11-20T06:06:58.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.935+0000 7f5895ef3700 1 -- 172.21.15.139:0/2191664297 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f589010ca90 con 0x7f58900f55a0 2021-11-20T06:06:58.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.935+0000 7f5895ef3700 1 -- 172.21.15.139:0/2191664297 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58900fdd30 con 0x7f58900f55a0 2021-11-20T06:06:58.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.936+0000 7f5877fff700 1 -- 172.21.15.139:0/2191664297 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f588001d420 con 0x7f58900f55a0 2021-11-20T06:06:58.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.936+0000 7f5877fff700 1 --2- 172.21.15.139:0/2191664297 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f587805c2a0 0x7f587805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:58.006 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.936+0000 7f5877fff700 1 -- 172.21.15.139:0/2191664297 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f58800781f0 con 0x7f58900f55a0 2021-11-20T06:06:58.007 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.936+0000 7f588effd700 1 --2- 172.21.15.139:0/2191664297 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f587805c2a0 0x7f587805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:58.007 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.937+0000 7f588effd700 1 --2- 172.21.15.139:0/2191664297 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f587805c2a0 0x7f587805e760 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f5884007fd0 tx=0x7f5884009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:58.007 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:57.940+0000 7f5877fff700 1 -- 172.21.15.139:0/2191664297 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f588004aa40 con 0x7f58900f55a0 2021-11-20T06:06:58.108 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:58.123+0000 7f5895ef3700 1 -- 172.21.15.139:0/2191664297 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f589010d3a0 con 0x7f58900f55a0 2021-11-20T06:06:58.108 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:58.124+0000 7f5877fff700 1 -- 172.21.15.139:0/2191664297 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2562 (secure 0 0 0) 0x7f588004e060 con 0x7f58900f55a0 2021-11-20T06:06:58.109 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:58.109 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":64,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:44.233143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14660":{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:58.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:58.125+0000 7f5895ef3700 1 -- 172.21.15.139:0/2191664297 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f587805c2a0 msgr2=0x7f587805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:58.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:58.125+0000 7f5895ef3700 1 --2- 172.21.15.139:0/2191664297 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f587805c2a0 0x7f587805e760 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f5884007fd0 tx=0x7f5884009040).stop 2021-11-20T06:06:58.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:58.125+0000 7f5895ef3700 1 -- 172.21.15.139:0/2191664297 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58900f55a0 msgr2=0x7f5890106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:58.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:58.125+0000 7f5895ef3700 1 --2- 172.21.15.139:0/2191664297 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58900f55a0 0x7f5890106eb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f5880009f50 tx=0x7f5880009b30).stop 2021-11-20T06:06:58.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:58.125+0000 7f5895ef3700 1 -- 172.21.15.139:0/2191664297 shutdown_connections 2021-11-20T06:06:58.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:58.125+0000 7f5895ef3700 1 --2- 172.21.15.139:0/2191664297 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f587805c2a0 0x7f587805e760 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:58.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:58.125+0000 7f5895ef3700 1 --2- 172.21.15.139:0/2191664297 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f58900f55a0 0x7f5890106eb0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:58.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:58.125+0000 7f5895ef3700 1 --2- 172.21.15.139:0/2191664297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58900f6840 0x7f58901073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:58.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:58.125+0000 7f5895ef3700 1 -- 172.21.15.139:0/2191664297 >> 172.21.15.139:0/2191664297 conn(0x7f58900f0ce0 msgr2=0x7f589005f450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:58.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:58.125+0000 7f5895ef3700 1 -- 172.21.15.139:0/2191664297 shutdown_connections 2021-11-20T06:06:58.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:58.125+0000 7f5895ef3700 1 -- 172.21.15.139:0/2191664297 wait complete. 2021-11-20T06:06:58.113 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 64 2021-11-20T06:06:58.122 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:44.233143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14660': {'gid': 14660, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6827/811251312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 811251312}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 811251312}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:58.122 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14660:up:rejoin 2021-11-20T06:06:58.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:58 smithi139 conmon[33501]: audit 2021-11-20T06:06:58.124594+0000 mon.smithi139 (mon. 2021-11-20T06:06:58.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:58 smithi139 conmon[33501]: 0) 756 : audit [DBG] from='client.? 172.21.15.139:0/2191664297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:58.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:58 smithi178 conmon[36600]: audit 2021-11-20T06:06:58.124594+0000 mon.smithi139 (mon.0) 756 2021-11-20T06:06:58.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:58 smithi178 conmon[36600]: : audit [DBG] from='client.? 172.21.15.139:0/2191664297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:06:59.124 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:06:59.255 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.268+0000 7f0cc3608700 1 -- 172.21.15.139:0/3914957038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cbc0f45c0 msgr2=0x7f0cbc0f49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:59.255 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.268+0000 7f0cc3608700 1 --2- 172.21.15.139:0/3914957038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cbc0f45c0 0x7f0cbc0f49e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f0cb80099e0 tx=0x7f0cb8004d10).stop 2021-11-20T06:06:59.256 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.268+0000 7f0cc3608700 1 -- 172.21.15.139:0/3914957038 shutdown_connections 2021-11-20T06:06:59.256 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.268+0000 7f0cc3608700 1 --2- 172.21.15.139:0/3914957038 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cbc0f5720 0x7f0cbc0f5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:59.256 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.268+0000 7f0cc3608700 1 --2- 172.21.15.139:0/3914957038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cbc0f45c0 0x7f0cbc0f49e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:59.256 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.268+0000 7f0cc3608700 1 -- 172.21.15.139:0/3914957038 >> 172.21.15.139:0/3914957038 conn(0x7f0cbc0efff0 msgr2=0x7f0cbc0f2430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:59.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.268+0000 7f0cc3608700 1 -- 172.21.15.139:0/3914957038 shutdown_connections 2021-11-20T06:06:59.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.268+0000 7f0cc3608700 1 -- 172.21.15.139:0/3914957038 wait complete. 2021-11-20T06:06:59.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.268+0000 7f0cc3608700 1 Processor -- start 2021-11-20T06:06:59.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.268+0000 7f0cc3608700 1 -- start start 2021-11-20T06:06:59.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.268+0000 7f0cc3608700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cbc0f45c0 0x7f0cbc10d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:59.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.268+0000 7f0cc3608700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cbc0f5720 0x7f0cbc10da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:59.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.268+0000 7f0cc3608700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc10e040 con 0x7f0cbc0f5720 2021-11-20T06:06:59.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.268+0000 7f0cc3608700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc10e180 con 0x7f0cbc0f45c0 2021-11-20T06:06:59.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.269+0000 7f0cc0ba3700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cbc0f5720 0x7f0cbc10da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:59.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.269+0000 7f0cc0ba3700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cbc0f5720 0x7f0cbc10da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58544/0 (socket says 172.21.15.139:58544) 2021-11-20T06:06:59.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.269+0000 7f0cc0ba3700 1 -- 172.21.15.139:0/70567103 learned_addr learned my addr 172.21.15.139:0/70567103 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:06:59.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.269+0000 7f0cc13a4700 1 --2- 172.21.15.139:0/70567103 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cbc0f45c0 0x7f0cbc10d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:59.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.269+0000 7f0cc0ba3700 1 -- 172.21.15.139:0/70567103 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cbc0f45c0 msgr2=0x7f0cbc10d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:59.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.269+0000 7f0cc0ba3700 1 --2- 172.21.15.139:0/70567103 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cbc0f45c0 0x7f0cbc10d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:59.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.269+0000 7f0cc0ba3700 1 -- 172.21.15.139:0/70567103 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb8009640 con 0x7f0cbc0f5720 2021-11-20T06:06:59.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.269+0000 7f0cc13a4700 1 --2- 172.21.15.139:0/70567103 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cbc0f45c0 0x7f0cbc10d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:06:59.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.269+0000 7f0cc0ba3700 1 --2- 172.21.15.139:0/70567103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cbc0f5720 0x7f0cbc10da90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f0cac002700 tx=0x7f0cac009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:59.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.269+0000 7f0cb1ffb700 1 -- 172.21.15.139:0/70567103 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cac009320 con 0x7f0cbc0f5720 2021-11-20T06:06:59.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.269+0000 7f0cc3608700 1 -- 172.21.15.139:0/70567103 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cbc112c20 con 0x7f0cbc0f5720 2021-11-20T06:06:59.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.269+0000 7f0cb1ffb700 1 -- 172.21.15.139:0/70567103 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cac00f040 con 0x7f0cbc0f5720 2021-11-20T06:06:59.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.269+0000 7f0cb1ffb700 1 -- 172.21.15.139:0/70567103 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cac014900 con 0x7f0cbc0f5720 2021-11-20T06:06:59.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.269+0000 7f0cc3608700 1 -- 172.21.15.139:0/70567103 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cbc113160 con 0x7f0cbc0f5720 2021-11-20T06:06:59.262 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.270+0000 7f0cb1ffb700 1 -- 172.21.15.139:0/70567103 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0cac014a60 con 0x7f0cbc0f5720 2021-11-20T06:06:59.262 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.270+0000 7f0cc3608700 1 -- 172.21.15.139:0/70567103 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cbc027a00 con 0x7f0cbc0f5720 2021-11-20T06:06:59.262 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.271+0000 7f0cb1ffb700 1 --2- 172.21.15.139:0/70567103 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0ca805c2a0 0x7f0ca805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:06:59.262 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.271+0000 7f0cb1ffb700 1 -- 172.21.15.139:0/70567103 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0cac0765a0 con 0x7f0cbc0f5720 2021-11-20T06:06:59.262 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.271+0000 7f0cc13a4700 1 --2- 172.21.15.139:0/70567103 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0ca805c2a0 0x7f0ca805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:06:59.263 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.271+0000 7f0cc13a4700 1 --2- 172.21.15.139:0/70567103 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0ca805c2a0 0x7f0ca805e760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f0cb800b620 tx=0x7f0cb8004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:06:59.263 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.274+0000 7f0cb1ffb700 1 -- 172.21.15.139:0/70567103 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0cac048eb0 con 0x7f0cbc0f5720 2021-11-20T06:06:59.443 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.458+0000 7f0cc3608700 1 -- 172.21.15.139:0/70567103 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cbc105430 con 0x7f0cbc0f5720 2021-11-20T06:06:59.443 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.458+0000 7f0cb1ffb700 1 -- 172.21.15.139:0/70567103 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2562 (secure 0 0 0) 0x7f0cac04c4d0 con 0x7f0cbc0f5720 2021-11-20T06:06:59.444 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:06:59.444 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":64,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:44.233143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14660":{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:06:59.445 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.460+0000 7f0cc3608700 1 -- 172.21.15.139:0/70567103 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0ca805c2a0 msgr2=0x7f0ca805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:59.445 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.460+0000 7f0cc3608700 1 --2- 172.21.15.139:0/70567103 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0ca805c2a0 0x7f0ca805e760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f0cb800b620 tx=0x7f0cb8004d10).stop 2021-11-20T06:06:59.446 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.460+0000 7f0cc3608700 1 -- 172.21.15.139:0/70567103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cbc0f5720 msgr2=0x7f0cbc10da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:06:59.446 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.460+0000 7f0cc3608700 1 --2- 172.21.15.139:0/70567103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cbc0f5720 0x7f0cbc10da90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f0cac002700 tx=0x7f0cac009d80).stop 2021-11-20T06:06:59.446 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.460+0000 7f0cc3608700 1 -- 172.21.15.139:0/70567103 shutdown_connections 2021-11-20T06:06:59.446 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.460+0000 7f0cc3608700 1 --2- 172.21.15.139:0/70567103 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0ca805c2a0 0x7f0ca805e760 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:59.447 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.460+0000 7f0cc3608700 1 --2- 172.21.15.139:0/70567103 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cbc0f45c0 0x7f0cbc10d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:59.447 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.460+0000 7f0cc3608700 1 --2- 172.21.15.139:0/70567103 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0cbc0f5720 0x7f0cbc10da90 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:06:59.447 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.460+0000 7f0cc3608700 1 -- 172.21.15.139:0/70567103 >> 172.21.15.139:0/70567103 conn(0x7f0cbc0efff0 msgr2=0x7f0cbc0f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:06:59.447 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.460+0000 7f0cc3608700 1 -- 172.21.15.139:0/70567103 shutdown_connections 2021-11-20T06:06:59.448 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:06:59.460+0000 7f0cc3608700 1 -- 172.21.15.139:0/70567103 wait complete. 2021-11-20T06:06:59.448 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 64 2021-11-20T06:06:59.455 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:59 smithi139 conmon[33501]: cluster 2021-11- 2021-11-20T06:06:59.456 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:06:59 smithi139 conmon[33501]: 20T06:06:58.264560+0000 mgr.smithi139.dplyxn (mgr.14162) 295 : cluster [DBG] pgmap v267: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 341 B/s rd, 0 op/s 2021-11-20T06:06:59.457 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:44.233143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14660': {'gid': 14660, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6827/811251312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 811251312}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 811251312}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:06:59.457 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14660:up:rejoin 2021-11-20T06:06:59.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:06:59 smithi178 conmon[36600]: cluster 2021-11-20T06:06:58.264560+0000 mgr.smithi139.dplyxn (mgr.14162) 295 : cluster [DBG] pgmap v267: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 341 B/s rd, 0 op/s 2021-11-20T06:07:00.460 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:00.487 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:00 smithi139 conmon[33501]: audit 2021-11-20T06:06:59.459292+0000 mon.smithi139 (mon.0) 757 : audit [DBG] 2021-11-20T06:07:00.488 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:00 smithi139 conmon[33501]: from='client.? 172.21.15.139:0/70567103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:00.588 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.601+0000 7fc8fbfff700 1 -- 172.21.15.139:0/460111733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8fc0f6290 msgr2=0x7fc8fc0f66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:00.589 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.601+0000 7fc8fbfff700 1 --2- 172.21.15.139:0/460111733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8fc0f6290 0x7fc8fc0f66f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fc8f0004660 tx=0x7fc8f0009b30).stop 2021-11-20T06:07:00.589 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.601+0000 7fc8fbfff700 1 -- 172.21.15.139:0/460111733 shutdown_connections 2021-11-20T06:07:00.589 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.601+0000 7fc8fbfff700 1 --2- 172.21.15.139:0/460111733 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8fc0f6290 0x7fc8fc0f66f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:00.589 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.601+0000 7fc8fbfff700 1 --2- 172.21.15.139:0/460111733 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8fc0f7c80 0x7fc8fc0f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:00.590 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.601+0000 7fc8fbfff700 1 -- 172.21.15.139:0/460111733 >> 172.21.15.139:0/460111733 conn(0x7fc8fc0f1d10 msgr2=0x7fc8fc0f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:00.590 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.601+0000 7fc8fbfff700 1 -- 172.21.15.139:0/460111733 shutdown_connections 2021-11-20T06:07:00.590 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.601+0000 7fc8fbfff700 1 -- 172.21.15.139:0/460111733 wait complete. 2021-11-20T06:07:00.590 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.602+0000 7fc8fbfff700 1 Processor -- start 2021-11-20T06:07:00.590 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.602+0000 7fc8fbfff700 1 -- start start 2021-11-20T06:07:00.591 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.602+0000 7fc8fbfff700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8fc0f6290 0x7fc8fc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:00.591 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.602+0000 7fc8fbfff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8fc0f7c80 0x7fc8fc069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:00.591 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.602+0000 7fc8fbfff700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc06a380 con 0x7fc8fc0f7c80 2021-11-20T06:07:00.591 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.602+0000 7fc8fbfff700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc06a4c0 con 0x7fc8fc0f6290 2021-11-20T06:07:00.591 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.602+0000 7fc8fa7fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8fc0f7c80 0x7fc8fc069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:00.592 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.602+0000 7fc8fa7fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8fc0f7c80 0x7fc8fc069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58554/0 (socket says 172.21.15.139:58554) 2021-11-20T06:07:00.592 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.602+0000 7fc8fa7fc700 1 -- 172.21.15.139:0/3753735837 learned_addr learned my addr 172.21.15.139:0/3753735837 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:00.592 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.602+0000 7fc8faffd700 1 --2- 172.21.15.139:0/3753735837 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8fc0f6290 0x7fc8fc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:00.592 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.603+0000 7fc8fa7fc700 1 -- 172.21.15.139:0/3753735837 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8fc0f6290 msgr2=0x7fc8fc06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:00.593 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.603+0000 7fc8fa7fc700 1 --2- 172.21.15.139:0/3753735837 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8fc0f6290 0x7fc8fc06b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:00.593 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.603+0000 7fc8fa7fc700 1 -- 172.21.15.139:0/3753735837 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f0005040 con 0x7fc8fc0f7c80 2021-11-20T06:07:00.593 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.603+0000 7fc8fa7fc700 1 --2- 172.21.15.139:0/3753735837 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8fc0f7c80 0x7fc8fc069e20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fc8f0009990 tx=0x7fc8f0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:00.593 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.603+0000 7fc8e37fe700 1 -- 172.21.15.139:0/3753735837 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8f0018070 con 0x7fc8fc0f7c80 2021-11-20T06:07:00.593 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.603+0000 7fc8e37fe700 1 -- 172.21.15.139:0/3753735837 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8f0013410 con 0x7fc8fc0f7c80 2021-11-20T06:07:00.594 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.603+0000 7fc8fbfff700 1 -- 172.21.15.139:0/3753735837 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8fc06a740 con 0x7fc8fc0f7c80 2021-11-20T06:07:00.594 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.603+0000 7fc8e37fe700 1 -- 172.21.15.139:0/3753735837 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8f001c3c0 con 0x7fc8fc0f7c80 2021-11-20T06:07:00.594 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.603+0000 7fc8fbfff700 1 -- 172.21.15.139:0/3753735837 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8fc06aca0 con 0x7fc8fc0f7c80 2021-11-20T06:07:00.594 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.604+0000 7fc8fbfff700 1 -- 172.21.15.139:0/3753735837 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8fc045ad0 con 0x7fc8fc0f7c80 2021-11-20T06:07:00.595 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.604+0000 7fc8e37fe700 1 -- 172.21.15.139:0/3753735837 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc8f001d420 con 0x7fc8fc0f7c80 2021-11-20T06:07:00.596 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.605+0000 7fc8e37fe700 1 --2- 172.21.15.139:0/3753735837 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc8e405c340 0x7fc8e405e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:00.596 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.605+0000 7fc8e37fe700 1 -- 172.21.15.139:0/3753735837 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc8f0078060 con 0x7fc8fc0f7c80 2021-11-20T06:07:00.596 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.605+0000 7fc8faffd700 1 --2- 172.21.15.139:0/3753735837 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc8e405c340 0x7fc8e405e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:00.596 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.605+0000 7fc8faffd700 1 --2- 172.21.15.139:0/3753735837 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc8e405c340 0x7fc8e405e800 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fc8fc06b730 tx=0x7fc8ec00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:00.597 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.609+0000 7fc8e37fe700 1 -- 172.21.15.139:0/3753735837 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc8f00a42b0 con 0x7fc8fc0f7c80 2021-11-20T06:07:00.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:00 smithi178 conmon[36600]: audit 2021-11-20T06:06:59.459292+0000 mon.smithi139 (mon.0) 757 : audit [DBG] from='client.? 172.21.15.139:0/70567103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:00.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.791+0000 7fc8fbfff700 1 -- 172.21.15.139:0/3753735837 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8fc027a00 con 0x7fc8fc0f7c80 2021-11-20T06:07:00.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.791+0000 7fc8e37fe700 1 -- 172.21.15.139:0/3753735837 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2562 (secure 0 0 0) 0x7fc8f004a850 con 0x7fc8fc0f7c80 2021-11-20T06:07:00.777 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:00.777 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":64,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:44.233143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14660":{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:00.778 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.793+0000 7fc8fbfff700 1 -- 172.21.15.139:0/3753735837 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc8e405c340 msgr2=0x7fc8e405e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:00.778 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.793+0000 7fc8fbfff700 1 --2- 172.21.15.139:0/3753735837 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc8e405c340 0x7fc8e405e800 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fc8fc06b730 tx=0x7fc8ec00b040).stop 2021-11-20T06:07:00.779 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.793+0000 7fc8fbfff700 1 -- 172.21.15.139:0/3753735837 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8fc0f7c80 msgr2=0x7fc8fc069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:00.779 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.793+0000 7fc8fbfff700 1 --2- 172.21.15.139:0/3753735837 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8fc0f7c80 0x7fc8fc069e20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fc8f0009990 tx=0x7fc8f0009b30).stop 2021-11-20T06:07:00.779 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.793+0000 7fc8fbfff700 1 -- 172.21.15.139:0/3753735837 shutdown_connections 2021-11-20T06:07:00.779 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.793+0000 7fc8fbfff700 1 --2- 172.21.15.139:0/3753735837 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc8e405c340 0x7fc8e405e800 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:00.780 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.793+0000 7fc8fbfff700 1 --2- 172.21.15.139:0/3753735837 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8fc0f6290 0x7fc8fc06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:00.780 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.793+0000 7fc8fbfff700 1 --2- 172.21.15.139:0/3753735837 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8fc0f7c80 0x7fc8fc069e20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:00.780 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.793+0000 7fc8fbfff700 1 -- 172.21.15.139:0/3753735837 >> 172.21.15.139:0/3753735837 conn(0x7fc8fc0f1d10 msgr2=0x7fc8fc0fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:00.780 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.793+0000 7fc8fbfff700 1 -- 172.21.15.139:0/3753735837 shutdown_connections 2021-11-20T06:07:00.781 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:00.793+0000 7fc8fbfff700 1 -- 172.21.15.139:0/3753735837 wait complete. 2021-11-20T06:07:00.781 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 64 2021-11-20T06:07:00.790 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:44.233143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14660': {'gid': 14660, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6827/811251312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 811251312}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 811251312}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:00.791 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14660:up:rejoin 2021-11-20T06:07:01.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:01 smithi139 conmon[33501]: cluster 2021-11-20T06:07:00.265054+0000 mgr.smithi139.dplyxn (mgr.14162) 296 : cluster [DBG] pgmap v268: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:07:01.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:01 smithi139 conmon[33501]: audit 2021-11-20T06:07:00.792153+0000 mon.smithi139 (mon.0) 758 : audit [DBG] from='client.? 172.21.15.139:0/3753735837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:01.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:01 smithi178 conmon[36600]: cluster 2021-11-20T06:07:00.265054+0000 mgr.smithi139.dplyxn (mgr.14162) 296 : cluster 2021-11-20T06:07:01.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:01 smithi178 conmon[36600]: [DBG] pgmap v268: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:07:01.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:01 smithi178 conmon[36600]: audit 2021-11-20T06:07:00.792153+0000 mon.smithi139 (mon.0) 758 : audit [DBG] from='client.? 172.21.15.139:0/3753735837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:01.792 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:01.921 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.933+0000 7f6a68ddb700 1 -- 172.21.15.139:0/2101061056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a640f7940 msgr2=0x7f6a640f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:01.921 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.933+0000 7f6a68ddb700 1 --2- 172.21.15.139:0/2101061056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a640f7940 0x7f6a640f7da0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f6a58004660 tx=0x7f6a58009b30).stop 2021-11-20T06:07:01.921 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.933+0000 7f6a68ddb700 1 -- 172.21.15.139:0/2101061056 shutdown_connections 2021-11-20T06:07:01.922 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.933+0000 7f6a68ddb700 1 --2- 172.21.15.139:0/2101061056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a640f7940 0x7f6a640f7da0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:01.922 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.933+0000 7f6a68ddb700 1 --2- 172.21.15.139:0/2101061056 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a640f66a0 0x7f6a640f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:01.922 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.933+0000 7f6a68ddb700 1 -- 172.21.15.139:0/2101061056 >> 172.21.15.139:0/2101061056 conn(0x7f6a640f1da0 msgr2=0x7f6a640f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:01.922 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.933+0000 7f6a68ddb700 1 -- 172.21.15.139:0/2101061056 shutdown_connections 2021-11-20T06:07:01.923 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.933+0000 7f6a68ddb700 1 -- 172.21.15.139:0/2101061056 wait complete. 2021-11-20T06:07:01.923 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a68ddb700 1 Processor -- start 2021-11-20T06:07:01.923 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a68ddb700 1 -- start start 2021-11-20T06:07:01.923 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a68ddb700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a640f66a0 0x7f6a64106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:01.924 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a68ddb700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a640f7940 0x7f6a641073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:01.924 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a68ddb700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a64107960 con 0x7f6a640f66a0 2021-11-20T06:07:01.924 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a68ddb700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a64107aa0 con 0x7f6a640f7940 2021-11-20T06:07:01.924 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a6259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a640f66a0 0x7f6a64106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:01.925 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a6259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a640f66a0 0x7f6a64106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58564/0 (socket says 172.21.15.139:58564) 2021-11-20T06:07:01.925 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a6259c700 1 -- 172.21.15.139:0/3466604533 learned_addr learned my addr 172.21.15.139:0/3466604533 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:01.925 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a61d9b700 1 --2- 172.21.15.139:0/3466604533 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a640f7940 0x7f6a641073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:01.925 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a6259c700 1 -- 172.21.15.139:0/3466604533 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a640f7940 msgr2=0x7f6a641073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:01.925 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a6259c700 1 --2- 172.21.15.139:0/3466604533 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a640f7940 0x7f6a641073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:01.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a6259c700 1 -- 172.21.15.139:0/3466604533 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a58005040 con 0x7f6a640f66a0 2021-11-20T06:07:01.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a61d9b700 1 --2- 172.21.15.139:0/3466604533 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a640f7940 0x7f6a641073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:07:01.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a6259c700 1 --2- 172.21.15.139:0/3466604533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a640f66a0 0x7f6a64106ec0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f6a5400c790 tx=0x7f6a54007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:01.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a52ffd700 1 -- 172.21.15.139:0/3466604533 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a5400e680 con 0x7f6a640f66a0 2021-11-20T06:07:01.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a52ffd700 1 -- 172.21.15.139:0/3466604533 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a5400ec90 con 0x7f6a640f66a0 2021-11-20T06:07:01.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a68ddb700 1 -- 172.21.15.139:0/3466604533 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6410c540 con 0x7f6a640f66a0 2021-11-20T06:07:01.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a52ffd700 1 -- 172.21.15.139:0/3466604533 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a54012cb0 con 0x7f6a640f66a0 2021-11-20T06:07:01.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.934+0000 7f6a68ddb700 1 -- 172.21.15.139:0/3466604533 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6410cb00 con 0x7f6a640f66a0 2021-11-20T06:07:01.928 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.935+0000 7f6a68ddb700 1 -- 172.21.15.139:0/3466604533 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a64027a00 con 0x7f6a640f66a0 2021-11-20T06:07:01.928 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.936+0000 7f6a52ffd700 1 -- 172.21.15.139:0/3466604533 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6a54010070 con 0x7f6a640f66a0 2021-11-20T06:07:01.928 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.936+0000 7f6a52ffd700 1 --2- 172.21.15.139:0/3466604533 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6a4c05c2a0 0x7f6a4c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:01.928 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.936+0000 7f6a52ffd700 1 -- 172.21.15.139:0/3466604533 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6a54076360 con 0x7f6a640f66a0 2021-11-20T06:07:01.928 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.936+0000 7f6a61d9b700 1 --2- 172.21.15.139:0/3466604533 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6a4c05c2a0 0x7f6a4c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:01.929 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.936+0000 7f6a61d9b700 1 --2- 172.21.15.139:0/3466604533 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6a4c05c2a0 0x7f6a4c05e760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f6a58013580 tx=0x7f6a58009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:01.929 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:01.940+0000 7f6a52ffd700 1 -- 172.21.15.139:0/3466604533 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a54048c70 con 0x7f6a640f66a0 2021-11-20T06:07:02.109 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:02.125+0000 7f6a68ddb700 1 -- 172.21.15.139:0/3466604533 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a640fee30 con 0x7f6a640f66a0 2021-11-20T06:07:02.110 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:02.125+0000 7f6a52ffd700 1 -- 172.21.15.139:0/3466604533 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+2562 (secure 0 0 0) 0x7f6a5404c290 con 0x7f6a640f66a0 2021-11-20T06:07:02.111 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:02.111 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":13},{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":64,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:06:44.233143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14660},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14660":{"gid":14660,"name":"cephfs.smithi139.riczqb","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.139:6827/811251312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":811251312},{"type":"v1","addr":"172.21.15.139:6827","nonce":811251312}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:02.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:02.127+0000 7f6a68ddb700 1 -- 172.21.15.139:0/3466604533 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6a4c05c2a0 msgr2=0x7f6a4c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:02.112 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:02.127+0000 7f6a68ddb700 1 --2- 172.21.15.139:0/3466604533 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6a4c05c2a0 0x7f6a4c05e760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f6a58013580 tx=0x7f6a58009b30).stop 2021-11-20T06:07:02.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:02.127+0000 7f6a68ddb700 1 -- 172.21.15.139:0/3466604533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a640f66a0 msgr2=0x7f6a64106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:02.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:02.127+0000 7f6a68ddb700 1 --2- 172.21.15.139:0/3466604533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a640f66a0 0x7f6a64106ec0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f6a5400c790 tx=0x7f6a54007d90).stop 2021-11-20T06:07:02.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:02.127+0000 7f6a68ddb700 1 -- 172.21.15.139:0/3466604533 shutdown_connections 2021-11-20T06:07:02.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:02.127+0000 7f6a68ddb700 1 --2- 172.21.15.139:0/3466604533 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6a4c05c2a0 0x7f6a4c05e760 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:02.114 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:02.127+0000 7f6a68ddb700 1 --2- 172.21.15.139:0/3466604533 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a640f66a0 0x7f6a64106ec0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:02.114 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:02.127+0000 7f6a68ddb700 1 --2- 172.21.15.139:0/3466604533 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a640f7940 0x7f6a641073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:02.114 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:02.127+0000 7f6a68ddb700 1 -- 172.21.15.139:0/3466604533 >> 172.21.15.139:0/3466604533 conn(0x7f6a640f1da0 msgr2=0x7f6a640fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:02.114 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:02.127+0000 7f6a68ddb700 1 -- 172.21.15.139:0/3466604533 shutdown_connections 2021-11-20T06:07:02.115 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:02.127+0000 7f6a68ddb700 1 -- 172.21.15.139:0/3466604533 wait complete. 2021-11-20T06:07:02.115 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 64 2021-11-20T06:07:02.123 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:06:44.233143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14660}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14660': {'gid': 14660, 'name': 'cephfs.smithi139.riczqb', 'rank': 0, 'incarnation': 62, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.139:6827/811251312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6826', 'nonce': 811251312}, {'type': 'v1', 'addr': '172.21.15.139:6827', 'nonce': 811251312}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:02.124 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14660:up:rejoin 2021-11-20T06:07:02.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:02 smithi178 conmon[36600]: audit 2021-11-20T06:07:02.125983+0000 mon.smithi139 (mon.0) 759 : audit [DBG] 2021-11-20T06:07:02.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:02 smithi178 conmon[36600]: from='client.? 172.21.15.139:0/3466604533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:02.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:02 smithi178 conmon[36600]: cluster 2021-11-20T06:07:02.224042+0000 mon.smithi139 (mon.0) 760 : cluster [WRN] Replacing daemon mds.cephfs.smithi139.riczqb as rank 0 with standby daemon mds.cephfs.smithi178.pskihy 2021-11-20T06:07:02.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:02 smithi178 conmon[36600]: cluster 2021-11-20T06:07:02.224081+0000 mon.smithi139 (mon.0 2021-11-20T06:07:02.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:02 smithi178 conmon[36600]: ) 761 : cluster [INF] MDS daemon mds.cephfs.smithi139.riczqb is removed because it is dead or otherwise unavailable. 2021-11-20T06:07:02.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:02 smithi178 conmon[36600]: cluster 2021-11-20T06:07:02.229961 2021-11-20T06:07:02.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:02 smithi178 conmon[36600]: +0000 mon.smithi139 (mon.0) 762 : cluster [DBG] osdmap e53: 6 total, 6 up, 6 in 2021-11-20T06:07:02.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:02 smithi178 conmon[36600]: cluster 2021-11-20T06:07:02.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:02 smithi178 conmon[36600]: 2021-11-20T06:07:02.235012+0000 2021-11-20T06:07:02.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:02 smithi178 conmon[36600]: mon.smithi139 (mon.0) 763 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:replay} 1 up:standby 2021-11-20T06:07:02.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:02 smithi139 conmon[33501]: audit 2021-11-20T06:07:02.125983+0000 mon.smithi139 (mon.0) 759 : audit 2021-11-20T06:07:02.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:02 smithi139 conmon[33501]: [DBG] from='client.? 172.21.15.139:0/3466604533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:02.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:02 smithi139 conmon[33501]: cluster 2021-11-20T06:07:02.224042+0000 mon.smithi139 (mon.0) 760 : cluster [WRN] Replacing daemon mds.cephfs.smithi139.riczqb as rank 0 with standby daemon mds.cephfs.smithi178.pskihy 2021-11-20T06:07:02.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:02 smithi139 conmon[33501]: cluster 2021- 2021-11-20T06:07:02.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:02 smithi139 conmon[33501]: 11-20T06:07:02.224081+0000 mon.smithi139 (mon.0) 761 : cluster [INF] MDS daemon mds.cephfs.smithi139.riczqb is removed because it is dead or otherwise unavailable. 2021-11-20T06:07:02.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:02 smithi139 conmon[33501]: cluster 2021-11- 2021-11-20T06:07:02.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:02 smithi139 conmon[33501]: 20T06:07:02.229961+0000 mon.smithi139 ( 2021-11-20T06:07:02.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:02 smithi139 conmon[33501]: mon.0) 762 : cluster [DBG] osdmap e53: 6 total, 6 up, 6 in 2021-11-20T06:07:02.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:02 smithi139 conmon[33501]: cluster 2021-11-20T06:07:02.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:02 smithi139 conmon[33501]: 2021-11- 2021-11-20T06:07:02.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:02 smithi139 conmon[33501]: 20T06:07:02.235012+0000 2021-11-20T06:07:02.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:02 smithi139 conmon[33501]: mon.smithi139 (mon.0) 763 : 2021-11-20T06:07:02.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:02 smithi139 conmon[33501]: cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:replay} 1 up:standby 2021-11-20T06:07:03.125 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:03.267 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.281+0000 7f6321503700 1 -- 172.21.15.139:0/2125321354 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f631c0f9840 msgr2=0x7f631c0fbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:03.267 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.281+0000 7f6321503700 1 --2- 172.21.15.139:0/2125321354 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f631c0f9840 0x7f631c0fbc50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f6310004660 tx=0x7f6310009b30).stop 2021-11-20T06:07:03.268 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.281+0000 7f6321503700 1 -- 172.21.15.139:0/2125321354 shutdown_connections 2021-11-20T06:07:03.268 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.281+0000 7f6321503700 1 --2- 172.21.15.139:0/2125321354 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f631c0f9840 0x7f631c0fbc50 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:03.268 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.281+0000 7f6321503700 1 --2- 172.21.15.139:0/2125321354 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f631c060100 0x7f631c0f92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:07:03.268 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.281+0000 7f6321503700 1 -- 172.21.15.139:0/2125321354 >> 172.21.15.139:0/2125321354 conn(0x7f631c0f0ce0 msgr2=0x7f631c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:03.269 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.281+0000 7f6321503700 1 -- 172.21.15.139:0/2125321354 shutdown_connections 2021-11-20T06:07:03.269 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.281+0000 7f6321503700 1 -- 172.21.15.139:0/2125321354 wait complete. 2021-11-20T06:07:03.269 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.281+0000 7f6321503700 1 Processor -- start 2021-11-20T06:07:03.269 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.281+0000 7f6321503700 1 -- start start 2021-11-20T06:07:03.269 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.281+0000 7f6321503700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f631c060100 0x7f631c106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:03.270 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.281+0000 7f6321503700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f631c0f9840 0x7f631c107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:03.270 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.281+0000 7f6321503700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631c1079c0 con 0x7f631c060100 2021-11-20T06:07:03.270 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.281+0000 7f6321503700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631c107b00 con 0x7f631c0f9840 2021-11-20T06:07:03.270 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.282+0000 7f631affd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f631c060100 0x7f631c106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:03.271 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.282+0000 7f631affd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f631c060100 0x7f631c106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58576/0 (socket says 172.21.15.139:58576) 2021-11-20T06:07:03.271 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.282+0000 7f631affd700 1 -- 172.21.15.139:0/2556232056 learned_addr learned my addr 172.21.15.139:0/2556232056 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:03.271 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.282+0000 7f631a7fc700 1 --2- 172.21.15.139:0/2556232056 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f631c0f9840 0x7f631c107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:03.271 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.282+0000 7f631affd700 1 -- 172.21.15.139:0/2556232056 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f631c0f9840 msgr2=0x7f631c107380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:03.271 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.282+0000 7f631affd700 1 --2- 172.21.15.139:0/2556232056 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f631c0f9840 0x7f631c107380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:03.272 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.282+0000 7f631affd700 1 -- 172.21.15.139:0/2556232056 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6310005040 con 0x7f631c060100 2021-11-20T06:07:03.272 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.282+0000 7f631affd700 1 --2- 172.21.15.139:0/2556232056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f631c060100 0x7f631c106e90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f630c00b410 tx=0x7f630c006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:03.272 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.282+0000 7f63037fe700 1 -- 172.21.15.139:0/2556232056 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f630c0125f0 con 0x7f631c060100 2021-11-20T06:07:03.272 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.282+0000 7f63037fe700 1 -- 172.21.15.139:0/2556232056 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f630c00be80 con 0x7f631c060100 2021-11-20T06:07:03.272 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.282+0000 7f6321503700 1 -- 172.21.15.139:0/2556232056 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631c10c5a0 con 0x7f631c060100 2021-11-20T06:07:03.273 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.282+0000 7f63037fe700 1 -- 172.21.15.139:0/2556232056 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f630c002ac0 con 0x7f631c060100 2021-11-20T06:07:03.273 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.282+0000 7f6321503700 1 -- 172.21.15.139:0/2556232056 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f631c10cb60 con 0x7f631c060100 2021-11-20T06:07:03.274 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.283+0000 7f6321503700 1 -- 172.21.15.139:0/2556232056 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f631c027a00 con 0x7f631c060100 2021-11-20T06:07:03.274 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.283+0000 7f63037fe700 1 -- 172.21.15.139:0/2556232056 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f630c00f030 con 0x7f631c060100 2021-11-20T06:07:03.275 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.284+0000 7f63037fe700 1 --2- 172.21.15.139:0/2556232056 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f630405c2a0 0x7f630405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:03.275 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.284+0000 7f63037fe700 1 -- 172.21.15.139:0/2556232056 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f630c0767d0 con 0x7f631c060100 2021-11-20T06:07:03.275 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.284+0000 7f631a7fc700 1 --2- 172.21.15.139:0/2556232056 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f630405c2a0 0x7f630405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:03.275 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.284+0000 7f631a7fc700 1 --2- 172.21.15.139:0/2556232056 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f630405c2a0 0x7f630405e760 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f6310009d60 tx=0x7f6310009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:03.276 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.288+0000 7f63037fe700 1 -- 172.21.15.139:0/2556232056 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f630c049090 con 0x7f631c060100 2021-11-20T06:07:03.458 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.473+0000 7f6321503700 1 -- 172.21.15.139:0/2556232056 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f631c0feeb0 con 0x7f631c060100 2021-11-20T06:07:03.459 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.474+0000 7f63037fe700 1 -- 172.21.15.139:0/2556232056 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2191 (secure 0 0 0) 0x7f630c04c6b0 con 0x7f631c060100 2021-11-20T06:07:03.459 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:03.460 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":66,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:03.233445+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24255":{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":65,"state":"up:reconnect","state_seq":53,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:03.461 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.476+0000 7f6321503700 1 -- 172.21.15.139:0/2556232056 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f630405c2a0 msgr2=0x7f630405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:03.462 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.476+0000 7f6321503700 1 --2- 172.21.15.139:0/2556232056 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f630405c2a0 0x7f630405e760 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f6310009d60 tx=0x7f6310009b30).stop 2021-11-20T06:07:03.462 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.476+0000 7f6321503700 1 -- 172.21.15.139:0/2556232056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f631c060100 msgr2=0x7f631c106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:03.462 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.476+0000 7f6321503700 1 --2- 172.21.15.139:0/2556232056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f631c060100 0x7f631c106e90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f630c00b410 tx=0x7f630c006ef0).stop 2021-11-20T06:07:03.462 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.476+0000 7f6321503700 1 -- 172.21.15.139:0/2556232056 shutdown_connections 2021-11-20T06:07:03.463 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.476+0000 7f6321503700 1 --2- 172.21.15.139:0/2556232056 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f630405c2a0 0x7f630405e760 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:03.463 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.476+0000 7f6321503700 1 --2- 172.21.15.139:0/2556232056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f631c060100 0x7f631c106e90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:03.463 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.476+0000 7f6321503700 1 --2- 172.21.15.139:0/2556232056 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f631c0f9840 0x7f631c107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:03.463 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.476+0000 7f6321503700 1 -- 172.21.15.139:0/2556232056 >> 172.21.15.139:0/2556232056 conn(0x7f631c0f0ce0 msgr2=0x7f631c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:03.463 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.476+0000 7f6321503700 1 -- 172.21.15.139:0/2556232056 shutdown_connections 2021-11-20T06:07:03.464 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:03.476+0000 7f6321503700 1 -- 172.21.15.139:0/2556232056 wait complete. 2021-11-20T06:07:03.464 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 66 2021-11-20T06:07:03.471 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:03 smithi139 conmon[33501]: cluster 2021-11-20T06:07:02.265408+0000 mgr.smithi139.dplyxn (mgr.14162) 297 : cluster [DBG] pgmap v270: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:07:03.472 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:03 smithi139 conmon[33501]: 2021-11-20T06:07:03.472 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:03 smithi139 conmon[33501]: cluster 2021-11-20T06:07:03.236506+0000 mon.smithi139 (mon.0) 764 : cluster 2021-11-20T06:07:03.472 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:03 smithi139 conmon[33501]: [DBG] mds.? [v2:172.21.15.178:6826/1921979486,v1:172.21.15.178:6827/1921979486] up:reconnect 2021-11-20T06:07:03.472 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:03 smithi139 conmon[33501]: cluster 2021-11-20T 2021-11-20T06:07:03.473 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:03 smithi139 conmon[33501]: 06:07:03.236572+0000 mon.smithi139 ( 2021-11-20T06:07:03.473 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:03 smithi139 conmon[33501]: mon.0) 765 : cluster 2021-11-20T06:07:03.473 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:03 smithi139 conmon[33501]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:reconnect} 1 up:standby 2021-11-20T06:07:03.474 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:03.233445+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24255}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24255': {'gid': 24255, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 65, 'state': 'up:reconnect', 'state_seq': 53, 'addr': '172.21.15.178:6827/1921979486', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 1921979486}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 1921979486}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:03.475 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24255:up:reconnect 2021-11-20T06:07:03.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:03 smithi178 conmon[36600]: cluster 2021-11-20T06:07:02.265408+0000 mgr.smithi139.dplyxn (mgr.14162) 297 : cluster [DBG] pgmap v270: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:07:03.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:03 smithi178 conmon[36600]: cluster 2021-11-20T06:07:03.236506+0000 mon.smithi139 (mon.0) 764 : cluster [DBG] mds.? [v2:172.21.15.178:6826/1921979486,v1:172.21.15.178:6827/1921979486] up:reconnect 2021-11-20T06:07:03.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:03 smithi178 conmon[36600]: cluster 2021-11-20T06:07:03.236572+0000 2021-11-20T06:07:03.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:03 smithi178 conmon[36600]: mon.smithi139 (mon.0) 765 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:reconnect} 1 up:standby 2021-11-20T06:07:04.477 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:04.505 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:04 smithi139 conmon[33501]: cluster 2021-11-20T06:07:03.257094+0000 mds.cephfs.smithi178.pskihy (mds.0) 1 : cluster 2021-11-20T06:07:04.506 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:04 smithi139 conmon[33501]: [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0219996 2021-11-20T06:07:04.506 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:04 smithi139 conmon[33501]: audit 2021-11-20T06:07:03.474867+0000 mon.smithi139 (mon.0) 766 : audit 2021-11-20T06:07:04.506 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:04 smithi139 conmon[33501]: [DBG] from='client.? 172.21.15.139:0/2556232056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:04.507 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:04 smithi139 conmon[33501]: cluster 2021-11-20T06:07:04.507 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:04 smithi139 conmon[33501]: 2021-11-20T 2021-11-20T06:07:04.507 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:04 smithi139 conmon[33501]: 06:07:04.239604+0000 mon.smithi139 (mon 2021-11-20T06:07:04.507 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:04 smithi139 conmon[33501]: .0) 767 : cluster [DBG] mds.? [v2:172.21.15.178:6826/1921979486,v1:172.21.15.178:6827/1921979486] up:rejoin 2021-11-20T06:07:04.508 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:04 smithi139 conmon[33501]: 2021-11-20T06:07:04.508 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:04 smithi139 conmon[33501]: cluster 2021- 2021-11-20T06:07:04.508 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:04 smithi139 conmon[33501]: 11-20T06:07:04.239732+0000 2021-11-20T06:07:04.508 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:04 smithi139 conmon[33501]: mon.smithi139 (mon.0) 768 : cluster [DBG] 2021-11-20T06:07:04.508 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:04 smithi139 conmon[33501]: fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:rejoin} 1 up:standby 2021-11-20T06:07:04.607 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.620+0000 7f75a92ff700 1 -- 172.21.15.139:0/1537565776 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a40f6840 msgr2=0x7f75a40f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:04.607 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.620+0000 7f75a92ff700 1 --2- 172.21.15.139:0/1537565776 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a40f6840 0x7f75a40f6ca0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f758c004660 tx=0x7f758c009b30).stop 2021-11-20T06:07:04.607 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.620+0000 7f75a92ff700 1 -- 172.21.15.139:0/1537565776 shutdown_connections 2021-11-20T06:07:04.608 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.620+0000 7f75a92ff700 1 --2- 172.21.15.139:0/1537565776 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a40f6840 0x7f75a40f6ca0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:04.608 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.620+0000 7f75a92ff700 1 --2- 172.21.15.139:0/1537565776 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a40f55a0 0x7f75a40f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:04.608 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.620+0000 7f75a92ff700 1 -- 172.21.15.139:0/1537565776 >> 172.21.15.139:0/1537565776 conn(0x7f75a40f0ce0 msgr2=0x7f75a40f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:04.608 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.620+0000 7f75a92ff700 1 -- 172.21.15.139:0/1537565776 shutdown_connections 2021-11-20T06:07:04.608 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.620+0000 7f75a92ff700 1 -- 172.21.15.139:0/1537565776 wait complete. 2021-11-20T06:07:04.609 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.620+0000 7f75a92ff700 1 Processor -- start 2021-11-20T06:07:04.609 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a92ff700 1 -- start start 2021-11-20T06:07:04.609 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a92ff700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a40f55a0 0x7f75a4106e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:04.609 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a92ff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a40f6840 0x7f75a4107330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:04.610 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a92ff700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a41078e0 con 0x7f75a40f6840 2021-11-20T06:07:04.610 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a92ff700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a4107a20 con 0x7f75a40f55a0 2021-11-20T06:07:04.610 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a27fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a40f6840 0x7f75a4107330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:04.610 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a27fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a40f6840 0x7f75a4107330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58590/0 (socket says 172.21.15.139:58590) 2021-11-20T06:07:04.611 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a2ffd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a40f55a0 0x7f75a4106e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:04.611 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a27fc700 1 -- 172.21.15.139:0/1777063427 learned_addr learned my addr 172.21.15.139:0/1777063427 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:04.611 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a27fc700 1 -- 172.21.15.139:0/1777063427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a40f55a0 msgr2=0x7f75a4106e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:04.611 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a27fc700 1 --2- 172.21.15.139:0/1777063427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a40f55a0 0x7f75a4106e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:04.611 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a27fc700 1 -- 172.21.15.139:0/1777063427 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758c005040 con 0x7f75a40f6840 2021-11-20T06:07:04.612 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a2ffd700 1 --2- 172.21.15.139:0/1777063427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a40f55a0 0x7f75a4106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-11-20T06:07:04.612 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a27fc700 1 --2- 172.21.15.139:0/1777063427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a40f6840 0x7f75a4107330 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f758c000c00 tx=0x7f758c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:04.612 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f759b7fe700 1 -- 172.21.15.139:0/1777063427 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f758c019070 con 0x7f75a40f6840 2021-11-20T06:07:04.612 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f759b7fe700 1 -- 172.21.15.139:0/1777063427 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f758c007e70 con 0x7f75a40f6840 2021-11-20T06:07:04.613 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a92ff700 1 -- 172.21.15.139:0/1777063427 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a410c460 con 0x7f75a40f6840 2021-11-20T06:07:04.613 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f759b7fe700 1 -- 172.21.15.139:0/1777063427 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f758c014c90 con 0x7f75a40f6840 2021-11-20T06:07:04.613 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.621+0000 7f75a92ff700 1 -- 172.21.15.139:0/1777063427 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a410ca20 con 0x7f75a40f6840 2021-11-20T06:07:04.613 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.622+0000 7f75a92ff700 1 -- 172.21.15.139:0/1777063427 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a40fdd30 con 0x7f75a40f6840 2021-11-20T06:07:04.614 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.623+0000 7f759b7fe700 1 -- 172.21.15.139:0/1777063427 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f758c025070 con 0x7f75a40f6840 2021-11-20T06:07:04.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.623+0000 7f759b7fe700 1 --2- 172.21.15.139:0/1777063427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f759005c2a0 0x7f759005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:04.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.623+0000 7f759b7fe700 1 -- 172.21.15.139:0/1777063427 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f758c077320 con 0x7f75a40f6840 2021-11-20T06:07:04.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.623+0000 7f75a2ffd700 1 --2- 172.21.15.139:0/1777063427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f759005c2a0 0x7f759005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:04.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.624+0000 7f75a2ffd700 1 --2- 172.21.15.139:0/1777063427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f759005c2a0 0x7f759005e760 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f7594004fd0 tx=0x7f759400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:04.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.627+0000 7f759b7fe700 1 -- 172.21.15.139:0/1777063427 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f758c049be0 con 0x7f75a40f6840 2021-11-20T06:07:04.659 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:04 smithi178 conmon[36600]: cluster 2021-11-20T06:07:03.257094+0000 mds.cephfs.smithi178.pskihy (mds.0) 1 : cluster [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0219996 2021-11-20T06:07:04.660 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:04 smithi178 conmon[36600]: 2021-11-20T06:07:04.660 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:04 smithi178 conmon[36600]: audit 2021-11-20T06:07:03.474867+0000 mon.smithi139 (mon.0) 766 : audit [DBG] from='client.? 172.21.15.139:0/2556232056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:04.660 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:04 smithi178 conmon[36600]: cluster 2021-11-20T06:07:04.239604+0000 mon.smithi139 (mon.0) 767 : cluster [DBG] mds.? [v2:172.21.15.178:6826/1921979486,v1:172.21.15.178:6827/1921979486] up:rejoin 2021-11-20T06:07:04.660 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:04 smithi178 conmon[36600]: cluster 2021-11-20T06:07:04.239732+0000 mon.smithi139 (mon.0) 768 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:rejoin} 1 up:standby 2021-11-20T06:07:04.802 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.817+0000 7f75a92ff700 1 -- 172.21.15.139:0/1777063427 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75a410d060 con 0x7f75a40f6840 2021-11-20T06:07:04.802 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.818+0000 7f759b7fe700 1 -- 172.21.15.139:0/1777063427 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f758c04d200 con 0x7f75a40f6840 2021-11-20T06:07:04.803 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:04.803 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":67,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:04.234595+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24255":{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":65,"state":"up:rejoin","state_seq":54,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:04.805 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.819+0000 7f75a92ff700 1 -- 172.21.15.139:0/1777063427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f759005c2a0 msgr2=0x7f759005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:04.805 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.819+0000 7f75a92ff700 1 --2- 172.21.15.139:0/1777063427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f759005c2a0 0x7f759005e760 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f7594004fd0 tx=0x7f759400b040).stop 2021-11-20T06:07:04.805 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.819+0000 7f75a92ff700 1 -- 172.21.15.139:0/1777063427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a40f6840 msgr2=0x7f75a4107330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:04.805 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.819+0000 7f75a92ff700 1 --2- 172.21.15.139:0/1777063427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a40f6840 0x7f75a4107330 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f758c000c00 tx=0x7f758c00b040).stop 2021-11-20T06:07:04.806 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.819+0000 7f75a92ff700 1 -- 172.21.15.139:0/1777063427 shutdown_connections 2021-11-20T06:07:04.806 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.819+0000 7f75a92ff700 1 --2- 172.21.15.139:0/1777063427 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f759005c2a0 0x7f759005e760 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:04.806 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.819+0000 7f75a92ff700 1 --2- 172.21.15.139:0/1777063427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75a40f55a0 0x7f75a4106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:04.806 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.819+0000 7f75a92ff700 1 --2- 172.21.15.139:0/1777063427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f75a40f6840 0x7f75a4107330 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:04.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.819+0000 7f75a92ff700 1 -- 172.21.15.139:0/1777063427 >> 172.21.15.139:0/1777063427 conn(0x7f75a40f0ce0 msgr2=0x7f75a405fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:04.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.819+0000 7f75a92ff700 1 -- 172.21.15.139:0/1777063427 shutdown_connections 2021-11-20T06:07:04.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:04.820+0000 7f75a92ff700 1 -- 172.21.15.139:0/1777063427 wait complete. 2021-11-20T06:07:04.807 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 67 2021-11-20T06:07:04.816 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:04.234595+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24255}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24255': {'gid': 24255, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 65, 'state': 'up:rejoin', 'state_seq': 54, 'addr': '172.21.15.178:6827/1921979486', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 1921979486}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 1921979486}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:04.817 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24255:up:rejoin 2021-11-20T06:07:05.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:05 smithi139 conmon[33501]: cluster 2021-11-20T06:07:04.266039+0000 mgr.smithi139.dplyxn (mgr.14162) 298 : cluster [DBG] pgmap v271: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T06:07:05.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:05 smithi139 conmon[33501]: audit 2021-11-20T06:07:04.818633+0000 mon.smithi139 (mon.0) 769 : audit [DBG] from='client.? 172.21.15.139:0/1777063427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:05.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:05 smithi178 conmon[36600]: cluster 2021-11-20T06:07:04.266039+0000 mgr.smithi139.dplyxn (mgr.14162) 298 : cluster [DBG] pgmap v271: 65 pgs: 65 active+clean; 2.5 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 204 B/s rd, 0 op/s 2021-11-20T06:07:05.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:05 smithi178 conmon[36600]: audit 2021-11-20T06:07:04.818633+0000 2021-11-20T06:07:05.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:05 smithi178 conmon[36600]: mon.smithi139 (mon.0) 769 : audit [DBG] from='client.? 172.21.15.139:0/1777063427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:05.818 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:05.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.959+0000 7fd64aebf700 1 -- 172.21.15.139:0/366094239 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6440f55a0 msgr2=0x7fd6440f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:05.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.959+0000 7fd64aebf700 1 --2- 172.21.15.139:0/366094239 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6440f55a0 0x7fd6440f59c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fd6340099e0 tx=0x7fd634004d10).stop 2021-11-20T06:07:05.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.959+0000 7fd64aebf700 1 -- 172.21.15.139:0/366094239 shutdown_connections 2021-11-20T06:07:05.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.959+0000 7fd64aebf700 1 --2- 172.21.15.139:0/366094239 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6440f6840 0x7fd6440f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:05.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.959+0000 7fd64aebf700 1 --2- 172.21.15.139:0/366094239 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6440f55a0 0x7fd6440f59c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:05.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.959+0000 7fd64aebf700 1 -- 172.21.15.139:0/366094239 >> 172.21.15.139:0/366094239 conn(0x7fd6440f0ce0 msgr2=0x7fd6440f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:05.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.959+0000 7fd64aebf700 1 -- 172.21.15.139:0/366094239 shutdown_connections 2021-11-20T06:07:05.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.959+0000 7fd64aebf700 1 -- 172.21.15.139:0/366094239 wait complete. 2021-11-20T06:07:05.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.959+0000 7fd64aebf700 1 Processor -- start 2021-11-20T06:07:05.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.959+0000 7fd64aebf700 1 -- start start 2021-11-20T06:07:05.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd64aebf700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6440f55a0 0x7fd644106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:05.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd64aebf700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6440f6840 0x7fd6441074a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:05.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd64aebf700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd644107a50 con 0x7fd6440f55a0 2021-11-20T06:07:05.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd64aebf700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd644107b90 con 0x7fd6440f6840 2021-11-20T06:07:05.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd648c5b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6440f55a0 0x7fd644106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:05.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd648c5b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6440f55a0 0x7fd644106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58602/0 (socket says 172.21.15.139:58602) 2021-11-20T06:07:05.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd648c5b700 1 -- 172.21.15.139:0/4036643410 learned_addr learned my addr 172.21.15.139:0/4036643410 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:05.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd643fff700 1 --2- 172.21.15.139:0/4036643410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6440f6840 0x7fd6441074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:05.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd648c5b700 1 -- 172.21.15.139:0/4036643410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6440f6840 msgr2=0x7fd6441074a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:05.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd648c5b700 1 --2- 172.21.15.139:0/4036643410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6440f6840 0x7fd6441074a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:05.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd648c5b700 1 -- 172.21.15.139:0/4036643410 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd634009640 con 0x7fd6440f55a0 2021-11-20T06:07:05.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd643fff700 1 --2- 172.21.15.139:0/4036643410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6440f6840 0x7fd6441074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:07:05.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd648c5b700 1 --2- 172.21.15.139:0/4036643410 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6440f55a0 0x7fd644106fb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fd634007830 tx=0x7fd634004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:05.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd6417fa700 1 -- 172.21.15.139:0/4036643410 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd63400e030 con 0x7fd6440f55a0 2021-11-20T06:07:05.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd6417fa700 1 -- 172.21.15.139:0/4036643410 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd63401d8d0 con 0x7fd6440f55a0 2021-11-20T06:07:05.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd64aebf700 1 -- 172.21.15.139:0/4036643410 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64410c5d0 con 0x7fd6440f55a0 2021-11-20T06:07:05.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd6417fa700 1 -- 172.21.15.139:0/4036643410 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd634018070 con 0x7fd6440f55a0 2021-11-20T06:07:05.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.960+0000 7fd64aebf700 1 -- 172.21.15.139:0/4036643410 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64410cb10 con 0x7fd6440f55a0 2021-11-20T06:07:05.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.961+0000 7fd64aebf700 1 -- 172.21.15.139:0/4036643410 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6440fdd30 con 0x7fd6440f55a0 2021-11-20T06:07:05.954 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.962+0000 7fd6417fa700 1 -- 172.21.15.139:0/4036643410 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd63401da40 con 0x7fd6440f55a0 2021-11-20T06:07:05.954 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.962+0000 7fd6417fa700 1 --2- 172.21.15.139:0/4036643410 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd63005c2a0 0x7fd63005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:05.954 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.962+0000 7fd6417fa700 1 -- 172.21.15.139:0/4036643410 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fd634078120 con 0x7fd6440f55a0 2021-11-20T06:07:05.954 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.962+0000 7fd643fff700 1 --2- 172.21.15.139:0/4036643410 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd63005c2a0 0x7fd63005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:05.955 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.962+0000 7fd643fff700 1 --2- 172.21.15.139:0/4036643410 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd63005c2a0 0x7fd63005e760 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd62c0041d0 tx=0x7fd62c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:05.955 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:05.966+0000 7fd6417fa700 1 -- 172.21.15.139:0/4036643410 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd63404aa60 con 0x7fd6440f55a0 2021-11-20T06:07:06.137 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:06.153+0000 7fd64aebf700 1 -- 172.21.15.139:0/4036643410 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd64410d1a0 con 0x7fd6440f55a0 2021-11-20T06:07:06.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:06.153+0000 7fd6417fa700 1 -- 172.21.15.139:0/4036643410 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2188 (secure 0 0 0) 0x7fd63404e080 con 0x7fd6440f55a0 2021-11-20T06:07:06.138 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:06.138 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":67,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:04.234595+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24255":{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":65,"state":"up:rejoin","state_seq":54,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:06.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:06.155+0000 7fd64aebf700 1 -- 172.21.15.139:0/4036643410 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd63005c2a0 msgr2=0x7fd63005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:06.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:06.155+0000 7fd64aebf700 1 --2- 172.21.15.139:0/4036643410 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd63005c2a0 0x7fd63005e760 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd62c0041d0 tx=0x7fd62c009040).stop 2021-11-20T06:07:06.141 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:06.155+0000 7fd64aebf700 1 -- 172.21.15.139:0/4036643410 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6440f55a0 msgr2=0x7fd644106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:06.141 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:06.155+0000 7fd64aebf700 1 --2- 172.21.15.139:0/4036643410 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6440f55a0 0x7fd644106fb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fd634007830 tx=0x7fd634004d10).stop 2021-11-20T06:07:06.141 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:06.155+0000 7fd64aebf700 1 -- 172.21.15.139:0/4036643410 shutdown_connections 2021-11-20T06:07:06.141 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:06.155+0000 7fd64aebf700 1 --2- 172.21.15.139:0/4036643410 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd63005c2a0 0x7fd63005e760 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:06.142 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:06.155+0000 7fd64aebf700 1 --2- 172.21.15.139:0/4036643410 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6440f55a0 0x7fd644106fb0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:06.142 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:06.155+0000 7fd64aebf700 1 --2- 172.21.15.139:0/4036643410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6440f6840 0x7fd6441074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:06.142 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:06.155+0000 7fd64aebf700 1 -- 172.21.15.139:0/4036643410 >> 172.21.15.139:0/4036643410 conn(0x7fd6440f0ce0 msgr2=0x7fd64405fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:06.142 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:06.155+0000 7fd64aebf700 1 -- 172.21.15.139:0/4036643410 shutdown_connections 2021-11-20T06:07:06.143 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:06.155+0000 7fd64aebf700 1 -- 172.21.15.139:0/4036643410 wait complete. 2021-11-20T06:07:06.143 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 67 2021-11-20T06:07:06.151 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:04.234595+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24255}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24255': {'gid': 24255, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 65, 'state': 'up:rejoin', 'state_seq': 54, 'addr': '172.21.15.178:6827/1921979486', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 1921979486}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 1921979486}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:06.152 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24255:up:rejoin 2021-11-20T06:07:06.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:06 smithi139 conmon[33501]: audit 2021-11-20T06:07:06.154001+0000 mon.smithi139 (mon.0) 770 : audit [DBG] from='client.? 172.21.15.139:0/4036643410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:06.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:06 smithi139 conmon[33501]: 2021-11-20T06:07:06.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:06 smithi178 conmon[36600]: audit 2021-11-20T06:07:06.154001+0000 mon.smithi139 (mon.0) 770 : audit [DBG] 2021-11-20T06:07:06.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:06 smithi178 conmon[36600]: from='client.? 172.21.15.139:0/4036643410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:07.153 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:07.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.290+0000 7f9f1759e700 1 -- 172.21.15.139:0/2701717042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f180f7840 msgr2=0x7f9f180f7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:07.279 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.290+0000 7f9f1759e700 1 --2- 172.21.15.139:0/2701717042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f180f7840 0x7f9f180f7ca0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f9f00004660 tx=0x7f9f00009b30).stop 2021-11-20T06:07:07.280 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.290+0000 7f9f1759e700 1 -- 172.21.15.139:0/2701717042 shutdown_connections 2021-11-20T06:07:07.280 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.290+0000 7f9f1759e700 1 --2- 172.21.15.139:0/2701717042 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f180f7840 0x7f9f180f7ca0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:07.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.290+0000 7f9f1759e700 1 --2- 172.21.15.139:0/2701717042 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f180f65a0 0x7f9f180f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:07.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.290+0000 7f9f1759e700 1 -- 172.21.15.139:0/2701717042 >> 172.21.15.139:0/2701717042 conn(0x7f9f180f1ce0 msgr2=0x7f9f180f4120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:07.281 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.290+0000 7f9f1759e700 1 -- 172.21.15.139:0/2701717042 shutdown_connections 2021-11-20T06:07:07.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.290+0000 7f9f1759e700 1 -- 172.21.15.139:0/2701717042 wait complete. 2021-11-20T06:07:07.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.290+0000 7f9f1759e700 1 Processor -- start 2021-11-20T06:07:07.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f1759e700 1 -- start start 2021-11-20T06:07:07.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f1759e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f180f65a0 0x7f9f18106d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:07.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f1759e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f180f7840 0x7f9f18107230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:07.283 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f1759e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f18107830 con 0x7f9f180f65a0 2021-11-20T06:07:07.284 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f1759e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f18108040 con 0x7f9f180f7840 2021-11-20T06:07:07.284 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f1659c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f180f65a0 0x7f9f18106d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:07.285 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f1659c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f180f65a0 0x7f9f18106d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58612/0 (socket says 172.21.15.139:58612) 2021-11-20T06:07:07.285 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f1659c700 1 -- 172.21.15.139:0/1526828374 learned_addr learned my addr 172.21.15.139:0/1526828374 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:07.286 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f15d9b700 1 --2- 172.21.15.139:0/1526828374 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f180f7840 0x7f9f18107230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:07.286 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f1659c700 1 -- 172.21.15.139:0/1526828374 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f180f7840 msgr2=0x7f9f18107230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:07.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f1659c700 1 --2- 172.21.15.139:0/1526828374 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f180f7840 0x7f9f18107230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:07.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f1659c700 1 -- 172.21.15.139:0/1526828374 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f00005040 con 0x7f9f180f65a0 2021-11-20T06:07:07.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f15d9b700 1 --2- 172.21.15.139:0/1526828374 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f180f7840 0x7f9f18107230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:07:07.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f1659c700 1 --2- 172.21.15.139:0/1526828374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f180f65a0 0x7f9f18106d40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f9f0800eb70 tx=0x7f9f0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:07.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f0effd700 1 -- 172.21.15.139:0/1526828374 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f080092e0 con 0x7f9f180f65a0 2021-11-20T06:07:07.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f0effd700 1 -- 172.21.15.139:0/1526828374 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f0800f040 con 0x7f9f180f65a0 2021-11-20T06:07:07.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f1759e700 1 -- 172.21.15.139:0/1526828374 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f181082c0 con 0x7f9f180f65a0 2021-11-20T06:07:07.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f0effd700 1 -- 172.21.15.139:0/1526828374 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f0800c9a0 con 0x7f9f180f65a0 2021-11-20T06:07:07.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.291+0000 7f9f1759e700 1 -- 172.21.15.139:0/1526828374 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f18108880 con 0x7f9f180f65a0 2021-11-20T06:07:07.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.292+0000 7f9f1759e700 1 -- 172.21.15.139:0/1526828374 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f180f5990 con 0x7f9f180f65a0 2021-11-20T06:07:07.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.293+0000 7f9f0effd700 1 -- 172.21.15.139:0/1526828374 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9f0801b030 con 0x7f9f180f65a0 2021-11-20T06:07:07.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.293+0000 7f9f0effd700 1 --2- 172.21.15.139:0/1526828374 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9f0405c300 0x7f9f0405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:07.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.293+0000 7f9f0effd700 1 -- 172.21.15.139:0/1526828374 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f9f08076300 con 0x7f9f180f65a0 2021-11-20T06:07:07.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.293+0000 7f9f15d9b700 1 --2- 172.21.15.139:0/1526828374 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9f0405c300 0x7f9f0405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:07.291 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.293+0000 7f9f15d9b700 1 --2- 172.21.15.139:0/1526828374 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9f0405c300 0x7f9f0405e7c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f9f00000c00 tx=0x7f9f0000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:07.292 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.297+0000 7f9f0effd700 1 -- 172.21.15.139:0/1526828374 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9f08048c70 con 0x7f9f180f65a0 2021-11-20T06:07:07.466 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.480+0000 7f9f1759e700 1 -- 172.21.15.139:0/1526828374 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f180fee60 con 0x7f9f180f65a0 2021-11-20T06:07:07.467 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.481+0000 7f9f0effd700 1 -- 172.21.15.139:0/1526828374 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f9f080198a0 con 0x7f9f180f65a0 2021-11-20T06:07:07.467 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:07.467 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":67,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:04.234595+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24255":{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":65,"state":"up:rejoin","state_seq":54,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:07.468 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.482+0000 7f9f1759e700 1 -- 172.21.15.139:0/1526828374 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9f0405c300 msgr2=0x7f9f0405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:07.469 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.482+0000 7f9f1759e700 1 --2- 172.21.15.139:0/1526828374 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9f0405c300 0x7f9f0405e7c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f9f00000c00 tx=0x7f9f0000b040).stop 2021-11-20T06:07:07.469 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.482+0000 7f9f1759e700 1 -- 172.21.15.139:0/1526828374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f180f65a0 msgr2=0x7f9f18106d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:07.469 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.482+0000 7f9f1759e700 1 --2- 172.21.15.139:0/1526828374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f180f65a0 0x7f9f18106d40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f9f0800eb70 tx=0x7f9f0800b040).stop 2021-11-20T06:07:07.469 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.482+0000 7f9f1759e700 1 -- 172.21.15.139:0/1526828374 shutdown_connections 2021-11-20T06:07:07.470 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.482+0000 7f9f1759e700 1 --2- 172.21.15.139:0/1526828374 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9f0405c300 0x7f9f0405e7c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:07.470 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.482+0000 7f9f1759e700 1 --2- 172.21.15.139:0/1526828374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9f180f65a0 0x7f9f18106d40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:07.470 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.482+0000 7f9f1759e700 1 --2- 172.21.15.139:0/1526828374 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f180f7840 0x7f9f18107230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:07.470 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.482+0000 7f9f1759e700 1 -- 172.21.15.139:0/1526828374 >> 172.21.15.139:0/1526828374 conn(0x7f9f180f1ce0 msgr2=0x7f9f180fb290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:07.471 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.482+0000 7f9f1759e700 1 -- 172.21.15.139:0/1526828374 shutdown_connections 2021-11-20T06:07:07.471 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:07.482+0000 7f9f1759e700 1 -- 172.21.15.139:0/1526828374 wait complete. 2021-11-20T06:07:07.471 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 67 2021-11-20T06:07:07.478 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:07 smithi139 conmon[33501]: cluster 2021-11-20T06:07:06.266432+0000 mgr.smithi139.dplyxn (mgr.14162) 299 : cluster [DBG] pgmap v272: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2021-11-20T06:07:07.480 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:04.234595+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24255}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24255': {'gid': 24255, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 65, 'state': 'up:rejoin', 'state_seq': 54, 'addr': '172.21.15.178:6827/1921979486', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 1921979486}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 1921979486}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:07.480 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24255:up:rejoin 2021-11-20T06:07:07.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:07 smithi178 conmon[36600]: cluster 2021-11-20T06:07:06.266432+0000 mgr.smithi139.dplyxn (mgr.14162) 299 2021-11-20T06:07:07.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:07 smithi178 conmon[36600]: : cluster [DBG] pgmap v272: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 921 B/s rd, 0 op/s 2021-11-20T06:07:08.482 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:08.509 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:08 smithi139 conmon[33501]: audit 2021-11-20T06:07:07.482639+0000 mon.smithi139 (mon.0) 771 : audit [DBG] 2021-11-20T06:07:08.510 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:08 smithi139 conmon[33501]: from='client.? 172.21.15.139:0/1526828374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:08.611 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.623+0000 7f538e88b700 1 -- 172.21.15.139:0/1304736582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53880f55a0 msgr2=0x7f53880f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:08.612 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.623+0000 7f538e88b700 1 --2- 172.21.15.139:0/1304736582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53880f55a0 0x7f53880f59c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f5378004660 tx=0x7f5378009b30).stop 2021-11-20T06:07:08.612 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.623+0000 7f538e88b700 1 -- 172.21.15.139:0/1304736582 shutdown_connections 2021-11-20T06:07:08.612 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.623+0000 7f538e88b700 1 --2- 172.21.15.139:0/1304736582 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53880f6840 0x7f53880f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:08.612 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.623+0000 7f538e88b700 1 --2- 172.21.15.139:0/1304736582 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53880f55a0 0x7f53880f59c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:08.612 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.623+0000 7f538e88b700 1 -- 172.21.15.139:0/1304736582 >> 172.21.15.139:0/1304736582 conn(0x7f53880f0ce0 msgr2=0x7f53880f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:08.613 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.623+0000 7f538e88b700 1 -- 172.21.15.139:0/1304736582 shutdown_connections 2021-11-20T06:07:08.613 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.623+0000 7f538e88b700 1 -- 172.21.15.139:0/1304736582 wait complete. 2021-11-20T06:07:08.613 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.623+0000 7f538e88b700 1 Processor -- start 2021-11-20T06:07:08.613 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.624+0000 7f538e88b700 1 -- start start 2021-11-20T06:07:08.614 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.624+0000 7f538e88b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53880f55a0 0x7f538805ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:08.614 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.624+0000 7f538e88b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53880f6840 0x7f538810bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:08.614 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.624+0000 7f538e88b700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538810c1d0 con 0x7f53880f55a0 2021-11-20T06:07:08.614 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.624+0000 7f538e88b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538810c310 con 0x7f53880f6840 2021-11-20T06:07:08.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.624+0000 7f5387fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53880f55a0 0x7f538805ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:08.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.624+0000 7f5387fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53880f55a0 0x7f538805ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58624/0 (socket says 172.21.15.139:58624) 2021-11-20T06:07:08.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.624+0000 7f5387fff700 1 -- 172.21.15.139:0/3433584122 learned_addr learned my addr 172.21.15.139:0/3433584122 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:08.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.624+0000 7f53877fe700 1 --2- 172.21.15.139:0/3433584122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53880f6840 0x7f538810bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:08.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.624+0000 7f5387fff700 1 -- 172.21.15.139:0/3433584122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53880f6840 msgr2=0x7f538810bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:08.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.624+0000 7f5387fff700 1 --2- 172.21.15.139:0/3433584122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53880f6840 0x7f538810bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:08.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.624+0000 7f5387fff700 1 -- 172.21.15.139:0/3433584122 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5378005040 con 0x7f53880f55a0 2021-11-20T06:07:08.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.624+0000 7f5387fff700 1 --2- 172.21.15.139:0/3433584122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53880f55a0 0x7f538805ff00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f5378004e30 tx=0x7f5378009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:08.617 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.625+0000 7f5384ff9700 1 -- 172.21.15.139:0/3433584122 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5378018070 con 0x7f53880f55a0 2021-11-20T06:07:08.617 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.625+0000 7f538e88b700 1 -- 172.21.15.139:0/3433584122 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538810c510 con 0x7f53880f55a0 2021-11-20T06:07:08.617 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.625+0000 7f5384ff9700 1 -- 172.21.15.139:0/3433584122 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5378013410 con 0x7f53880f55a0 2021-11-20T06:07:08.617 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.625+0000 7f5384ff9700 1 -- 172.21.15.139:0/3433584122 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f537801c3c0 con 0x7f53880f55a0 2021-11-20T06:07:08.618 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.625+0000 7f538e88b700 1 -- 172.21.15.139:0/3433584122 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f538810ca40 con 0x7f53880f55a0 2021-11-20T06:07:08.618 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.625+0000 7f538e88b700 1 -- 172.21.15.139:0/3433584122 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5388046330 con 0x7f53880f55a0 2021-11-20T06:07:08.618 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.626+0000 7f5384ff9700 1 -- 172.21.15.139:0/3433584122 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f537801d420 con 0x7f53880f55a0 2021-11-20T06:07:08.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.626+0000 7f5384ff9700 1 --2- 172.21.15.139:0/3433584122 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f537005c2a0 0x7f537005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:08.626 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.626+0000 7f5384ff9700 1 -- 172.21.15.139:0/3433584122 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f5378077ec0 con 0x7f53880f55a0 2021-11-20T06:07:08.626 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.626+0000 7f53877fe700 1 --2- 172.21.15.139:0/3433584122 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f537005c2a0 0x7f537005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:08.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.627+0000 7f53877fe700 1 --2- 172.21.15.139:0/3433584122 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f537005c2a0 0x7f537005e760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f537c0096c0 tx=0x7f537c009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:08.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.630+0000 7f5384ff9700 1 -- 172.21.15.139:0/3433584122 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f537804a780 con 0x7f53880f55a0 2021-11-20T06:07:08.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:08 smithi178 conmon[36600]: audit 2021-11-20T06:07:07.482639+0000 mon.smithi139 (mon.0) 771 : audit [DBG] 2021-11-20T06:07:08.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:08 smithi178 conmon[36600]: from='client.? 172.21.15.139:0/1526828374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:08.803 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.817+0000 7f538e88b700 1 -- 172.21.15.139:0/3433584122 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5388027a00 con 0x7f53880f55a0 2021-11-20T06:07:08.804 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.818+0000 7f5384ff9700 1 -- 172.21.15.139:0/3433584122 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f537804dda0 con 0x7f53880f55a0 2021-11-20T06:07:08.804 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:08.804 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":67,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:04.234595+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24255":{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":65,"state":"up:rejoin","state_seq":54,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:08.806 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.819+0000 7f538e88b700 1 -- 172.21.15.139:0/3433584122 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f537005c2a0 msgr2=0x7f537005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:08.806 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.819+0000 7f538e88b700 1 --2- 172.21.15.139:0/3433584122 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f537005c2a0 0x7f537005e760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f537c0096c0 tx=0x7f537c009be0).stop 2021-11-20T06:07:08.806 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.819+0000 7f538e88b700 1 -- 172.21.15.139:0/3433584122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53880f55a0 msgr2=0x7f538805ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:08.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.819+0000 7f538e88b700 1 --2- 172.21.15.139:0/3433584122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53880f55a0 0x7f538805ff00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f5378004e30 tx=0x7f5378009b30).stop 2021-11-20T06:07:08.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.819+0000 7f538e88b700 1 -- 172.21.15.139:0/3433584122 shutdown_connections 2021-11-20T06:07:08.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.819+0000 7f538e88b700 1 --2- 172.21.15.139:0/3433584122 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f537005c2a0 0x7f537005e760 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:08.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.819+0000 7f538e88b700 1 --2- 172.21.15.139:0/3433584122 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53880f55a0 0x7f538805ff00 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:08.808 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.819+0000 7f538e88b700 1 --2- 172.21.15.139:0/3433584122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53880f6840 0x7f538810bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:08.808 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.819+0000 7f538e88b700 1 -- 172.21.15.139:0/3433584122 >> 172.21.15.139:0/3433584122 conn(0x7f53880f0ce0 msgr2=0x7f53880fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:08.808 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.820+0000 7f538e88b700 1 -- 172.21.15.139:0/3433584122 shutdown_connections 2021-11-20T06:07:08.808 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:08.820+0000 7f538e88b700 1 -- 172.21.15.139:0/3433584122 wait complete. 2021-11-20T06:07:08.809 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 67 2021-11-20T06:07:08.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:04.234595+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24255}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24255': {'gid': 24255, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 65, 'state': 'up:rejoin', 'state_seq': 54, 'addr': '172.21.15.178:6827/1921979486', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 1921979486}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 1921979486}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:08.819 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24255:up:rejoin 2021-11-20T06:07:09.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:09 smithi139 conmon[33501]: cluster 2021-11-20T06:07:08.266918+0000 mgr.smithi139.dplyxn (mgr.14162) 300 : cluster 2021-11-20T06:07:09.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:09 smithi139 conmon[33501]: [DBG] pgmap v273: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:09.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:09 smithi139 conmon[33501]: audit 2021-11-20T06:07:08.819687+0000 mon.smithi139 (mon.0) 772 : audit [DBG] from='client.? 172.21.15.139:0/3433584122' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:09.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:09 smithi178 conmon[36600]: cluster 2021-11-20T06:07:08.266918+0000 mgr.smithi139.dplyxn (mgr.14162) 300 : cluster 2021-11-20T06:07:09.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:09 smithi178 conmon[36600]: [DBG] pgmap v273: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:09.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:09 smithi178 conmon[36600]: audit 2021-11-20T06:07:08.819687+0000 mon.smithi139 (mon.0) 772 : audit [DBG] from='client.? 172.21.15.139:0/3433584122' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:09.821 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:09.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.959+0000 7f8987264700 1 -- 172.21.15.139:0/3354644421 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89800f6760 msgr2=0x7f89800f6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:09.947 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.959+0000 7f8987264700 1 --2- 172.21.15.139:0/3354644421 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89800f6760 0x7f89800f6bc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f8970004660 tx=0x7f8970009b30).stop 2021-11-20T06:07:09.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.959+0000 7f8987264700 1 -- 172.21.15.139:0/3354644421 shutdown_connections 2021-11-20T06:07:09.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.959+0000 7f8987264700 1 --2- 172.21.15.139:0/3354644421 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89800f6760 0x7f89800f6bc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:09.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.959+0000 7f8987264700 1 --2- 172.21.15.139:0/3354644421 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89800f54c0 0x7f89800f58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:09.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.959+0000 7f8987264700 1 -- 172.21.15.139:0/3354644421 >> 172.21.15.139:0/3354644421 conn(0x7f89800f0c00 msgr2=0x7f89800f3040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:09.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.959+0000 7f8987264700 1 -- 172.21.15.139:0/3354644421 shutdown_connections 2021-11-20T06:07:09.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.959+0000 7f8987264700 1 -- 172.21.15.139:0/3354644421 wait complete. 2021-11-20T06:07:09.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.960+0000 7f8987264700 1 Processor -- start 2021-11-20T06:07:09.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.960+0000 7f8987264700 1 -- start start 2021-11-20T06:07:09.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.960+0000 7f8987264700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89800f54c0 0x7f8980109a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:09.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.960+0000 7f8987264700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89800f6760 0x7f898010bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:09.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.960+0000 7f8987264700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8980060520 con 0x7f89800f54c0 2021-11-20T06:07:09.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.960+0000 7f8987264700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8980060660 con 0x7f89800f6760 2021-11-20T06:07:09.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.960+0000 7f8986262700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89800f54c0 0x7f8980109a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:09.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.960+0000 7f8986262700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89800f54c0 0x7f8980109a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58634/0 (socket says 172.21.15.139:58634) 2021-11-20T06:07:09.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.960+0000 7f8986262700 1 -- 172.21.15.139:0/1196265229 learned_addr learned my addr 172.21.15.139:0/1196265229 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:09.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.960+0000 7f8985a61700 1 --2- 172.21.15.139:0/1196265229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89800f6760 0x7f898010bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:09.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.960+0000 7f8986262700 1 -- 172.21.15.139:0/1196265229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89800f6760 msgr2=0x7f898010bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:09.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.960+0000 7f8986262700 1 --2- 172.21.15.139:0/1196265229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89800f6760 0x7f898010bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:09.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.960+0000 7f8986262700 1 -- 172.21.15.139:0/1196265229 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8970005040 con 0x7f89800f54c0 2021-11-20T06:07:09.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.960+0000 7f8986262700 1 --2- 172.21.15.139:0/1196265229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89800f54c0 0x7f8980109a40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f897c002700 tx=0x7f897c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:09.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.961+0000 7f8976ffd700 1 -- 172.21.15.139:0/1196265229 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f897c009320 con 0x7f89800f54c0 2021-11-20T06:07:09.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.961+0000 7f8976ffd700 1 -- 172.21.15.139:0/1196265229 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f897c00f040 con 0x7f89800f54c0 2021-11-20T06:07:09.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.961+0000 7f8987264700 1 -- 172.21.15.139:0/1196265229 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f898010c4f0 con 0x7f89800f54c0 2021-11-20T06:07:09.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.961+0000 7f8976ffd700 1 -- 172.21.15.139:0/1196265229 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f897c014900 con 0x7f89800f54c0 2021-11-20T06:07:09.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.961+0000 7f8987264700 1 -- 172.21.15.139:0/1196265229 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898010cab0 con 0x7f89800f54c0 2021-11-20T06:07:09.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.961+0000 7f8987264700 1 -- 172.21.15.139:0/1196265229 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8980045ad0 con 0x7f89800f54c0 2021-11-20T06:07:09.954 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.962+0000 7f8976ffd700 1 -- 172.21.15.139:0/1196265229 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f897c009480 con 0x7f89800f54c0 2021-11-20T06:07:09.955 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.962+0000 7f8976ffd700 1 --2- 172.21.15.139:0/1196265229 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f896c05c300 0x7f896c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:09.955 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.962+0000 7f8976ffd700 1 -- 172.21.15.139:0/1196265229 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f897c076610 con 0x7f89800f54c0 2021-11-20T06:07:09.955 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.962+0000 7f8985a61700 1 --2- 172.21.15.139:0/1196265229 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f896c05c300 0x7f896c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:09.955 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.963+0000 7f8985a61700 1 --2- 172.21.15.139:0/1196265229 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f896c05c300 0x7f896c05e7c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f8970004e30 tx=0x7f8970009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:09.956 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:09.966+0000 7f8976ffd700 1 -- 172.21.15.139:0/1196265229 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f897c048ed0 con 0x7f89800f54c0 2021-11-20T06:07:10.137 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:10.151+0000 7f8987264700 1 -- 172.21.15.139:0/1196265229 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89800fdd60 con 0x7f89800f54c0 2021-11-20T06:07:10.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:10.152+0000 7f8976ffd700 1 -- 172.21.15.139:0/1196265229 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f897c04c4f0 con 0x7f89800f54c0 2021-11-20T06:07:10.138 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:10.138 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":67,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:04.234595+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24255":{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":65,"state":"up:rejoin","state_seq":54,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:10.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:10.153+0000 7f8987264700 1 -- 172.21.15.139:0/1196265229 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f896c05c300 msgr2=0x7f896c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:10.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:10.153+0000 7f8987264700 1 --2- 172.21.15.139:0/1196265229 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f896c05c300 0x7f896c05e7c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f8970004e30 tx=0x7f8970009b30).stop 2021-11-20T06:07:10.140 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:10.153+0000 7f8987264700 1 -- 172.21.15.139:0/1196265229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89800f54c0 msgr2=0x7f8980109a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:10.141 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:10.153+0000 7f8987264700 1 --2- 172.21.15.139:0/1196265229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89800f54c0 0x7f8980109a40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f897c002700 tx=0x7f897c009d80).stop 2021-11-20T06:07:10.141 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:10.153+0000 7f8987264700 1 -- 172.21.15.139:0/1196265229 shutdown_connections 2021-11-20T06:07:10.141 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:10.153+0000 7f8987264700 1 --2- 172.21.15.139:0/1196265229 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f896c05c300 0x7f896c05e7c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:10.141 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:10.153+0000 7f8987264700 1 --2- 172.21.15.139:0/1196265229 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f89800f54c0 0x7f8980109a40 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:10.141 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:10.153+0000 7f8987264700 1 --2- 172.21.15.139:0/1196265229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89800f6760 0x7f898010bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:10.142 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:10.153+0000 7f8987264700 1 -- 172.21.15.139:0/1196265229 >> 172.21.15.139:0/1196265229 conn(0x7f89800f0c00 msgr2=0x7f89800fa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:10.142 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:10.154+0000 7f8987264700 1 -- 172.21.15.139:0/1196265229 shutdown_connections 2021-11-20T06:07:10.142 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:10.154+0000 7f8987264700 1 -- 172.21.15.139:0/1196265229 wait complete. 2021-11-20T06:07:10.142 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 67 2021-11-20T06:07:10.151 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:04.234595+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24255}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24255': {'gid': 24255, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 65, 'state': 'up:rejoin', 'state_seq': 54, 'addr': '172.21.15.178:6827/1921979486', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 1921979486}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 1921979486}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:10.151 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24255:up:rejoin 2021-11-20T06:07:10.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:10 smithi139 conmon[33501]: audit 2021-11-20T06:07:10.153672+0000 mon.smithi139 (mon. 2021-11-20T06:07:10.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:10 smithi139 conmon[33501]: 0) 773 : audit [DBG] from='client.? 172.21.15.139:0/1196265229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:10.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:10 smithi178 conmon[36600]: audit 2021-11-20T06:07:10.153672+0000 mon.smithi139 (mon.0) 773 : audit [DBG] from='client.? 172.21.15.139:0/1196265229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:11.153 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:11.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.294+0000 7fa62c59c700 1 -- 172.21.15.139:0/202092570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6240f55c0 msgr2=0x7fa6240f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:11.282 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.294+0000 7fa62c59c700 1 --2- 172.21.15.139:0/202092570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6240f55c0 0x7fa6240f59e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fa620004660 tx=0x7fa620009b30).stop 2021-11-20T06:07:11.284 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.294+0000 7fa62c59c700 1 -- 172.21.15.139:0/202092570 shutdown_connections 2021-11-20T06:07:11.285 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.294+0000 7fa62c59c700 1 --2- 172.21.15.139:0/202092570 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa6240f6860 0x7fa6240f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:11.286 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.294+0000 7fa62c59c700 1 --2- 172.21.15.139:0/202092570 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6240f55c0 0x7fa6240f59e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:11.286 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.294+0000 7fa62c59c700 1 -- 172.21.15.139:0/202092570 >> 172.21.15.139:0/202092570 conn(0x7fa6240f0d00 msgr2=0x7fa6240f3140 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:11.286 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.294+0000 7fa62c59c700 1 -- 172.21.15.139:0/202092570 shutdown_connections 2021-11-20T06:07:11.286 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.294+0000 7fa62c59c700 1 -- 172.21.15.139:0/202092570 wait complete. 2021-11-20T06:07:11.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.295+0000 7fa62c59c700 1 Processor -- start 2021-11-20T06:07:11.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.295+0000 7fa62c59c700 1 -- start start 2021-11-20T06:07:11.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.295+0000 7fa62c59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6240f55c0 0x7fa624105db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:11.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.295+0000 7fa62c59c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa6240f6860 0x7fa6241062a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:11.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.295+0000 7fa62c59c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa624106850 con 0x7fa6240f55c0 2021-11-20T06:07:11.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.295+0000 7fa62c59c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa624106990 con 0x7fa6240f6860 2021-11-20T06:07:11.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.295+0000 7fa62a338700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6240f55c0 0x7fa624105db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:11.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.295+0000 7fa62a338700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6240f55c0 0x7fa624105db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58644/0 (socket says 172.21.15.139:58644) 2021-11-20T06:07:11.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.295+0000 7fa62a338700 1 -- 172.21.15.139:0/1433712505 learned_addr learned my addr 172.21.15.139:0/1433712505 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:11.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.295+0000 7fa629b37700 1 --2- 172.21.15.139:0/1433712505 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa6240f6860 0x7fa6241062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:11.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.295+0000 7fa62a338700 1 -- 172.21.15.139:0/1433712505 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa6240f6860 msgr2=0x7fa6241062a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:11.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.295+0000 7fa62a338700 1 --2- 172.21.15.139:0/1433712505 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa6240f6860 0x7fa6241062a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:11.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.295+0000 7fa62a338700 1 -- 172.21.15.139:0/1433712505 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa620005040 con 0x7fa6240f55c0 2021-11-20T06:07:11.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.295+0000 7fa62a338700 1 --2- 172.21.15.139:0/1433712505 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6240f55c0 0x7fa624105db0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fa620004e30 tx=0x7fa620009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:11.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.296+0000 7fa61affd700 1 -- 172.21.15.139:0/1433712505 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa620018070 con 0x7fa6240f55c0 2021-11-20T06:07:11.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.296+0000 7fa61affd700 1 -- 172.21.15.139:0/1433712505 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa620013410 con 0x7fa6240f55c0 2021-11-20T06:07:11.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.296+0000 7fa62c59c700 1 -- 172.21.15.139:0/1433712505 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa62410b3d0 con 0x7fa6240f55c0 2021-11-20T06:07:11.291 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.296+0000 7fa61affd700 1 -- 172.21.15.139:0/1433712505 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa62001c3c0 con 0x7fa6240f55c0 2021-11-20T06:07:11.291 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.296+0000 7fa62c59c700 1 -- 172.21.15.139:0/1433712505 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa62410b990 con 0x7fa6240f55c0 2021-11-20T06:07:11.291 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.297+0000 7fa62c59c700 1 -- 172.21.15.139:0/1433712505 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa624027a00 con 0x7fa6240f55c0 2021-11-20T06:07:11.293 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.297+0000 7fa61affd700 1 -- 172.21.15.139:0/1433712505 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa62001d420 con 0x7fa6240f55c0 2021-11-20T06:07:11.293 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.297+0000 7fa61affd700 1 --2- 172.21.15.139:0/1433712505 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa61005c2a0 0x7fa61005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:11.293 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.298+0000 7fa61affd700 1 -- 172.21.15.139:0/1433712505 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fa620077e90 con 0x7fa6240f55c0 2021-11-20T06:07:11.293 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.298+0000 7fa629b37700 1 --2- 172.21.15.139:0/1433712505 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa61005c2a0 0x7fa61005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:11.293 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.298+0000 7fa629b37700 1 --2- 172.21.15.139:0/1433712505 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa61005c2a0 0x7fa61005e760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fa614004a50 tx=0x7fa61400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:11.294 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.301+0000 7fa61affd700 1 -- 172.21.15.139:0/1433712505 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa62004a7d0 con 0x7fa6240f55c0 2021-11-20T06:07:11.471 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.486+0000 7fa62c59c700 1 -- 172.21.15.139:0/1433712505 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6240fdd50 con 0x7fa6240f55c0 2021-11-20T06:07:11.472 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.486+0000 7fa61affd700 1 -- 172.21.15.139:0/1433712505 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2188 (secure 0 0 0) 0x7fa62004ddf0 con 0x7fa6240f55c0 2021-11-20T06:07:11.473 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:11.473 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":67,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:04.234595+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24255":{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":65,"state":"up:rejoin","state_seq":54,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:11.474 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.488+0000 7fa62c59c700 1 -- 172.21.15.139:0/1433712505 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa61005c2a0 msgr2=0x7fa61005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:11.474 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.488+0000 7fa62c59c700 1 --2- 172.21.15.139:0/1433712505 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa61005c2a0 0x7fa61005e760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fa614004a50 tx=0x7fa61400b040).stop 2021-11-20T06:07:11.475 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.488+0000 7fa62c59c700 1 -- 172.21.15.139:0/1433712505 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6240f55c0 msgr2=0x7fa624105db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:11.475 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.488+0000 7fa62c59c700 1 --2- 172.21.15.139:0/1433712505 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6240f55c0 0x7fa624105db0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fa620004e30 tx=0x7fa620009b30).stop 2021-11-20T06:07:11.475 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.488+0000 7fa62c59c700 1 -- 172.21.15.139:0/1433712505 shutdown_connections 2021-11-20T06:07:11.476 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.488+0000 7fa62c59c700 1 --2- 172.21.15.139:0/1433712505 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa61005c2a0 0x7fa61005e760 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:11.476 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.488+0000 7fa62c59c700 1 --2- 172.21.15.139:0/1433712505 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa6240f55c0 0x7fa624105db0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:11.476 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.488+0000 7fa62c59c700 1 --2- 172.21.15.139:0/1433712505 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa6240f6860 0x7fa6241062a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:11.476 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.488+0000 7fa62c59c700 1 -- 172.21.15.139:0/1433712505 >> 172.21.15.139:0/1433712505 conn(0x7fa6240f0d00 msgr2=0x7fa6240fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:11.477 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.488+0000 7fa62c59c700 1 -- 172.21.15.139:0/1433712505 shutdown_connections 2021-11-20T06:07:11.477 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:11.488+0000 7fa62c59c700 1 -- 172.21.15.139:0/1433712505 wait complete. 2021-11-20T06:07:11.477 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 67 2021-11-20T06:07:11.484 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:11 smithi139 conmon[33501]: cluster 2021-11-20T06:07:10.267324 2021-11-20T06:07:11.484 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:11 smithi139 conmon[33501]: +0000 mgr.smithi139.dplyxn (mgr.14162) 301 : cluster [DBG] pgmap v274: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:11.486 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:04.234595+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24255}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24255': {'gid': 24255, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 65, 'state': 'up:rejoin', 'state_seq': 54, 'addr': '172.21.15.178:6827/1921979486', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 1921979486}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 1921979486}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:11.486 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24255:up:rejoin 2021-11-20T06:07:11.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:11 smithi178 conmon[36600]: cluster 2021-11-20T06:07:10.267324+0000 mgr.smithi139.dplyxn (mgr.14162) 301 : cluster 2021-11-20T06:07:11.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:11 smithi178 conmon[36600]: [DBG] pgmap v274: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:12.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:12 smithi178 conmon[36600]: audit 2021-11-20T06:07:11.488337+0000 mon.smithi139 (mon.0) 774 : audit 2021-11-20T06:07:12.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:12 smithi178 conmon[36600]: [DBG] from='client.? 172.21.15.139:0/1433712505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:12.487 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:12.515 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:12 smithi139 conmon[33501]: audit 2021-11-20T06:07:11.488337+0000 mon.smithi139 (mon.0) 774 : audit [DBG] 2021-11-20T06:07:12.515 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:12 smithi139 conmon[33501]: from='client.? 172.21.15.139:0/1433712505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:12.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.627+0000 7ff651f72700 1 -- 172.21.15.139:0/1478666670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff64c0f7c80 msgr2=0x7ff64c0f80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:12.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.627+0000 7ff651f72700 1 --2- 172.21.15.139:0/1478666670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff64c0f7c80 0x7ff64c0f80a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7ff640004660 tx=0x7ff640009b30).stop 2021-11-20T06:07:12.615 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.627+0000 7ff651f72700 1 -- 172.21.15.139:0/1478666670 shutdown_connections 2021-11-20T06:07:12.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.627+0000 7ff651f72700 1 --2- 172.21.15.139:0/1478666670 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff64c0f6290 0x7ff64c0f66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:12.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.627+0000 7ff651f72700 1 --2- 172.21.15.139:0/1478666670 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff64c0f7c80 0x7ff64c0f80a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:12.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.627+0000 7ff651f72700 1 -- 172.21.15.139:0/1478666670 >> 172.21.15.139:0/1478666670 conn(0x7ff64c0f1d10 msgr2=0x7ff64c0f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:12.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.627+0000 7ff651f72700 1 -- 172.21.15.139:0/1478666670 shutdown_connections 2021-11-20T06:07:12.616 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.627+0000 7ff651f72700 1 -- 172.21.15.139:0/1478666670 wait complete. 2021-11-20T06:07:12.617 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.628+0000 7ff651f72700 1 Processor -- start 2021-11-20T06:07:12.617 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.628+0000 7ff651f72700 1 -- start start 2021-11-20T06:07:12.617 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.628+0000 7ff651f72700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff64c0f6290 0x7ff64c106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:12.617 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.628+0000 7ff651f72700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff64c0f7c80 0x7ff64c1073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:12.618 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.628+0000 7ff651f72700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c107960 con 0x7ff64c0f6290 2021-11-20T06:07:12.618 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.628+0000 7ff651f72700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c108160 con 0x7ff64c0f7c80 2021-11-20T06:07:12.618 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.628+0000 7ff650f70700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff64c0f6290 0x7ff64c106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:12.618 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.628+0000 7ff650f70700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff64c0f6290 0x7ff64c106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58656/0 (socket says 172.21.15.139:58656) 2021-11-20T06:07:12.618 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.628+0000 7ff650f70700 1 -- 172.21.15.139:0/2728006559 learned_addr learned my addr 172.21.15.139:0/2728006559 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:12.619 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.629+0000 7ff650f70700 1 -- 172.21.15.139:0/2728006559 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff64c0f7c80 msgr2=0x7ff64c1073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:12.619 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.629+0000 7ff650f70700 1 --2- 172.21.15.139:0/2728006559 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff64c0f7c80 0x7ff64c1073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:07:12.619 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.629+0000 7ff650f70700 1 -- 172.21.15.139:0/2728006559 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff640005040 con 0x7ff64c0f6290 2021-11-20T06:07:12.619 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.629+0000 7ff650f70700 1 --2- 172.21.15.139:0/2728006559 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff64c0f6290 0x7ff64c106ec0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7ff640009f50 tx=0x7ff640009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:12.620 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.629+0000 7ff6497fa700 1 -- 172.21.15.139:0/2728006559 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff640018070 con 0x7ff64c0f6290 2021-11-20T06:07:12.620 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.629+0000 7ff6497fa700 1 -- 172.21.15.139:0/2728006559 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff640013410 con 0x7ff64c0f6290 2021-11-20T06:07:12.620 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.629+0000 7ff6497fa700 1 -- 172.21.15.139:0/2728006559 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff64001c620 con 0x7ff64c0f6290 2021-11-20T06:07:12.620 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.629+0000 7ff651f72700 1 -- 172.21.15.139:0/2728006559 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64c1083e0 con 0x7ff64c0f6290 2021-11-20T06:07:12.620 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.629+0000 7ff651f72700 1 -- 172.21.15.139:0/2728006559 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64c108940 con 0x7ff64c0f6290 2021-11-20T06:07:12.621 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.630+0000 7ff651f72700 1 -- 172.21.15.139:0/2728006559 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64c0f59c0 con 0x7ff64c0f6290 2021-11-20T06:07:12.622 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.630+0000 7ff6497fa700 1 -- 172.21.15.139:0/2728006559 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff64001d420 con 0x7ff64c0f6290 2021-11-20T06:07:12.622 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.630+0000 7ff6497fa700 1 --2- 172.21.15.139:0/2728006559 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff63c05c300 0x7ff63c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:12.622 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.631+0000 7ff6497fa700 1 -- 172.21.15.139:0/2728006559 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7ff640078130 con 0x7ff64c0f6290 2021-11-20T06:07:12.622 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.631+0000 7ff64bfff700 1 --2- 172.21.15.139:0/2728006559 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff63c05c300 0x7ff63c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:12.623 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.631+0000 7ff64bfff700 1 --2- 172.21.15.139:0/2728006559 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff63c05c300 0x7ff63c05e7c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7ff638004870 tx=0x7ff6380079c0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:12.623 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.634+0000 7ff6497fa700 1 -- 172.21.15.139:0/2728006559 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff64004a9f0 con 0x7ff64c0f6290 2021-11-20T06:07:12.804 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.819+0000 7ff651f72700 1 -- 172.21.15.139:0/2728006559 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff64c045ad0 con 0x7ff64c0f6290 2021-11-20T06:07:12.805 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.819+0000 7ff6497fa700 1 -- 172.21.15.139:0/2728006559 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2188 (secure 0 0 0) 0x7ff64c045ad0 con 0x7ff64c0f6290 2021-11-20T06:07:12.805 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:12.806 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":67,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:04.234595+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24255":{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":65,"state":"up:rejoin","state_seq":54,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:12.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.821+0000 7ff651f72700 1 -- 172.21.15.139:0/2728006559 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff63c05c300 msgr2=0x7ff63c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:12.807 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.821+0000 7ff651f72700 1 --2- 172.21.15.139:0/2728006559 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff63c05c300 0x7ff63c05e7c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7ff638004870 tx=0x7ff6380079c0).stop 2021-11-20T06:07:12.808 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.821+0000 7ff651f72700 1 -- 172.21.15.139:0/2728006559 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff64c0f6290 msgr2=0x7ff64c106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:12.808 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.821+0000 7ff651f72700 1 --2- 172.21.15.139:0/2728006559 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff64c0f6290 0x7ff64c106ec0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7ff640009f50 tx=0x7ff640009b30).stop 2021-11-20T06:07:12.808 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.821+0000 7ff651f72700 1 -- 172.21.15.139:0/2728006559 shutdown_connections 2021-11-20T06:07:12.808 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.821+0000 7ff651f72700 1 --2- 172.21.15.139:0/2728006559 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff63c05c300 0x7ff63c05e7c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:12.808 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.821+0000 7ff651f72700 1 --2- 172.21.15.139:0/2728006559 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff64c0f6290 0x7ff64c106ec0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:12.809 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.821+0000 7ff651f72700 1 --2- 172.21.15.139:0/2728006559 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff64c0f7c80 0x7ff64c1073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:07:12.809 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.821+0000 7ff651f72700 1 -- 172.21.15.139:0/2728006559 >> 172.21.15.139:0/2728006559 conn(0x7ff64c0f1d10 msgr2=0x7ff64c0fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:12.809 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.821+0000 7ff651f72700 1 -- 172.21.15.139:0/2728006559 shutdown_connections 2021-11-20T06:07:12.809 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:12.821+0000 7ff651f72700 1 -- 172.21.15.139:0/2728006559 wait complete. 2021-11-20T06:07:12.810 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 67 2021-11-20T06:07:12.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:04.234595+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24255}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24255': {'gid': 24255, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 65, 'state': 'up:rejoin', 'state_seq': 54, 'addr': '172.21.15.178:6827/1921979486', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 1921979486}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 1921979486}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:12.819 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24255:up:rejoin 2021-11-20T06:07:13.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:13 smithi178 conmon[36600]: cluster 2021-11-20T06:07:12.267940+0000 mgr.smithi139.dplyxn (mgr.14162) 302 : cluster 2021-11-20T06:07:13.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:13 smithi178 conmon[36600]: [DBG] pgmap v275: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:13.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:13 smithi178 conmon[36600]: audit 2021-11-20T06:07:12.821020+0000 mon.smithi139 (mon.0) 775 : audit [DBG] from='client.? 172.21.15.139:0/2728006559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:13.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:13 smithi139 conmon[33501]: cluster 2021-11-20T06:07:12.267940+0000 mgr.smithi139.dplyxn (mgr.14162) 302 : cluster [DBG] 2021-11-20T06:07:13.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:13 smithi139 conmon[33501]: pgmap v275: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:13.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:13 smithi139 conmon[33501]: audit 2021-11-20T06:07:12.821020+0000 mon.smithi139 (mon.0) 775 : audit [DBG] from='client.? 172.21.15.139:0/2728006559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:13.820 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:13.948 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.960+0000 7f6c69208700 1 -- 172.21.15.139:0/1229755510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c640f55a0 msgr2=0x7f6c640f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:13.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.960+0000 7f6c69208700 1 --2- 172.21.15.139:0/1229755510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c640f55a0 0x7f6c640f59c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f6c54004660 tx=0x7f6c54009b30).stop 2021-11-20T06:07:13.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.960+0000 7f6c69208700 1 -- 172.21.15.139:0/1229755510 shutdown_connections 2021-11-20T06:07:13.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.960+0000 7f6c69208700 1 --2- 172.21.15.139:0/1229755510 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6c640f6840 0x7f6c640f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:13.949 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.960+0000 7f6c69208700 1 --2- 172.21.15.139:0/1229755510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c640f55a0 0x7f6c640f59c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:13.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.960+0000 7f6c69208700 1 -- 172.21.15.139:0/1229755510 >> 172.21.15.139:0/1229755510 conn(0x7f6c640f0ce0 msgr2=0x7f6c640f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:13.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.960+0000 7f6c69208700 1 -- 172.21.15.139:0/1229755510 shutdown_connections 2021-11-20T06:07:13.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.960+0000 7f6c69208700 1 -- 172.21.15.139:0/1229755510 wait complete. 2021-11-20T06:07:13.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.960+0000 7f6c69208700 1 Processor -- start 2021-11-20T06:07:13.950 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.960+0000 7f6c69208700 1 -- start start 2021-11-20T06:07:13.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c69208700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6c640f55a0 0x7f6c64106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:13.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c69208700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c640f6840 0x7f6c641073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:13.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c69208700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c64107950 con 0x7f6c640f6840 2021-11-20T06:07:13.951 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c69208700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c64107a90 con 0x7f6c640f55a0 2021-11-20T06:07:13.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c6259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c640f6840 0x7f6c641073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:13.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c6259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c640f6840 0x7f6c641073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58668/0 (socket says 172.21.15.139:58668) 2021-11-20T06:07:13.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c6259c700 1 -- 172.21.15.139:0/1666861552 learned_addr learned my addr 172.21.15.139:0/1666861552 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:13.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c62d9d700 1 --2- 172.21.15.139:0/1666861552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6c640f55a0 0x7f6c64106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:13.952 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c6259c700 1 -- 172.21.15.139:0/1666861552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6c640f55a0 msgr2=0x7f6c64106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:13.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c6259c700 1 --2- 172.21.15.139:0/1666861552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6c640f55a0 0x7f6c64106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:13.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c6259c700 1 -- 172.21.15.139:0/1666861552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c54005040 con 0x7f6c640f6840 2021-11-20T06:07:13.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c6259c700 1 --2- 172.21.15.139:0/1666861552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c640f6840 0x7f6c641073a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f6c580026d0 tx=0x7f6c58009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:13.953 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c4b7fe700 1 -- 172.21.15.139:0/1666861552 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c58009320 con 0x7f6c640f6840 2021-11-20T06:07:13.954 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c69208700 1 -- 172.21.15.139:0/1666861552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6410c530 con 0x7f6c640f6840 2021-11-20T06:07:13.954 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c4b7fe700 1 -- 172.21.15.139:0/1666861552 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c5800f040 con 0x7f6c640f6840 2021-11-20T06:07:13.954 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c4b7fe700 1 -- 172.21.15.139:0/1666861552 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c580147e0 con 0x7f6c640f6840 2021-11-20T06:07:13.954 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.961+0000 7f6c69208700 1 -- 172.21.15.139:0/1666861552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6410caf0 con 0x7f6c640f6840 2021-11-20T06:07:13.954 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.962+0000 7f6c69208700 1 -- 172.21.15.139:0/1666861552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c640fdd30 con 0x7f6c640f6840 2021-11-20T06:07:13.955 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.962+0000 7f6c4b7fe700 1 -- 172.21.15.139:0/1666861552 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6c58009480 con 0x7f6c640f6840 2021-11-20T06:07:13.955 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.963+0000 7f6c4b7fe700 1 --2- 172.21.15.139:0/1666861552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6c4c05c2a0 0x7f6c4c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:13.955 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.963+0000 7f6c4b7fe700 1 -- 172.21.15.139:0/1666861552 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f6c58076a00 con 0x7f6c640f6840 2021-11-20T06:07:13.955 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.963+0000 7f6c62d9d700 1 --2- 172.21.15.139:0/1666861552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6c4c05c2a0 0x7f6c4c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:13.956 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.963+0000 7f6c62d9d700 1 --2- 172.21.15.139:0/1666861552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6c4c05c2a0 0x7f6c4c05e760 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f6c54009d60 tx=0x7f6c54009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:13.956 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:13.967+0000 7f6c4b7fe700 1 -- 172.21.15.139:0/1666861552 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c580492c0 con 0x7f6c640f6840 2021-11-20T06:07:14.134 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:14.149+0000 7f6c69208700 1 -- 172.21.15.139:0/1666861552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c6410d1f0 con 0x7f6c640f6840 2021-11-20T06:07:14.135 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:14.149+0000 7f6c4b7fe700 1 -- 172.21.15.139:0/1666861552 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f6c5804c8e0 con 0x7f6c640f6840 2021-11-20T06:07:14.135 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:14.136 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":67,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:04.234595+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24255":{"gid":24255,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":65,"state":"up:rejoin","state_seq":54,"addr":"172.21.15.178:6827/1921979486","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1921979486},{"type":"v1","addr":"172.21.15.178:6827","nonce":1921979486}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:14.137 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:14.151+0000 7f6c69208700 1 -- 172.21.15.139:0/1666861552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6c4c05c2a0 msgr2=0x7f6c4c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:14.137 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:14.151+0000 7f6c69208700 1 --2- 172.21.15.139:0/1666861552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6c4c05c2a0 0x7f6c4c05e760 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f6c54009d60 tx=0x7f6c54009b30).stop 2021-11-20T06:07:14.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:14.151+0000 7f6c69208700 1 -- 172.21.15.139:0/1666861552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c640f6840 msgr2=0x7f6c641073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:14.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:14.151+0000 7f6c69208700 1 --2- 172.21.15.139:0/1666861552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c640f6840 0x7f6c641073a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f6c580026d0 tx=0x7f6c58009de0).stop 2021-11-20T06:07:14.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:14.151+0000 7f6c69208700 1 -- 172.21.15.139:0/1666861552 shutdown_connections 2021-11-20T06:07:14.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:14.151+0000 7f6c69208700 1 --2- 172.21.15.139:0/1666861552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6c4c05c2a0 0x7f6c4c05e760 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:14.138 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:14.151+0000 7f6c69208700 1 --2- 172.21.15.139:0/1666861552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6c640f55a0 0x7f6c64106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:14.139 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:14.151+0000 7f6c69208700 1 --2- 172.21.15.139:0/1666861552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6c640f6840 0x7f6c641073a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:14.139 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:14.151+0000 7f6c69208700 1 -- 172.21.15.139:0/1666861552 >> 172.21.15.139:0/1666861552 conn(0x7f6c640f0ce0 msgr2=0x7f6c6405f450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:14.139 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:14.151+0000 7f6c69208700 1 -- 172.21.15.139:0/1666861552 shutdown_connections 2021-11-20T06:07:14.139 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:14.151+0000 7f6c69208700 1 -- 172.21.15.139:0/1666861552 wait complete. 2021-11-20T06:07:14.140 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 67 2021-11-20T06:07:14.148 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:04.234595+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24255}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24255': {'gid': 24255, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 65, 'state': 'up:rejoin', 'state_seq': 54, 'addr': '172.21.15.178:6827/1921979486', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 1921979486}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 1921979486}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:14.149 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24255:up:rejoin 2021-11-20T06:07:14.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:14 smithi139 conmon[33501]: audit 2021-11-20T06:07:14.151058+0000 mon.smithi139 (mon.0) 776 2021-11-20T06:07:14.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:14 smithi139 conmon[33501]: : audit [DBG] from='client.? 172.21.15.139:0/1666861552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:14.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:14 smithi178 conmon[36600]: audit 2021-11-20T06:07:14.151058+0000 mon.smithi139 (mon.0) 776 : audit 2021-11-20T06:07:14.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:14 smithi178 conmon[36600]: [DBG] from='client.? 172.21.15.139:0/1666861552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:15.150 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:15.285 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.295+0000 7fcb9163c700 1 -- 172.21.15.139:0/3061458189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb8c0f5580 msgr2=0x7fcb8c0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:15.286 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.295+0000 7fcb9163c700 1 --2- 172.21.15.139:0/3061458189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb8c0f5580 0x7fcb8c0f59a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fcb7c004660 tx=0x7fcb7c009b30).stop 2021-11-20T06:07:15.286 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.295+0000 7fcb9163c700 1 -- 172.21.15.139:0/3061458189 shutdown_connections 2021-11-20T06:07:15.286 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.295+0000 7fcb9163c700 1 --2- 172.21.15.139:0/3061458189 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb8c0f6820 0x7fcb8c0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:15.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.295+0000 7fcb9163c700 1 --2- 172.21.15.139:0/3061458189 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb8c0f5580 0x7fcb8c0f59a0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:15.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.295+0000 7fcb9163c700 1 -- 172.21.15.139:0/3061458189 >> 172.21.15.139:0/3061458189 conn(0x7fcb8c0f0ce0 msgr2=0x7fcb8c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:15.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.295+0000 7fcb9163c700 1 -- 172.21.15.139:0/3061458189 shutdown_connections 2021-11-20T06:07:15.287 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.295+0000 7fcb9163c700 1 -- 172.21.15.139:0/3061458189 wait complete. 2021-11-20T06:07:15.288 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.296+0000 7fcb9163c700 1 Processor -- start 2021-11-20T06:07:15.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.296+0000 7fcb9163c700 1 -- start start 2021-11-20T06:07:15.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.296+0000 7fcb9163c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb8c0f5580 0x7fcb8c060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:15.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.296+0000 7fcb9163c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb8c0f6820 0x7fcb8c10bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:15.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.296+0000 7fcb9163c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c10c190 con 0x7fcb8c0f6820 2021-11-20T06:07:15.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.296+0000 7fcb9163c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c10c2d0 con 0x7fcb8c0f5580 2021-11-20T06:07:15.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.296+0000 7fcb8a7fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb8c0f6820 0x7fcb8c10bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:15.291 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.296+0000 7fcb8a7fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb8c0f6820 0x7fcb8c10bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58678/0 (socket says 172.21.15.139:58678) 2021-11-20T06:07:15.291 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.296+0000 7fcb8a7fc700 1 -- 172.21.15.139:0/1567479898 learned_addr learned my addr 172.21.15.139:0/1567479898 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:15.291 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.296+0000 7fcb8affd700 1 --2- 172.21.15.139:0/1567479898 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb8c0f5580 0x7fcb8c060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:15.291 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.296+0000 7fcb8a7fc700 1 -- 172.21.15.139:0/1567479898 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb8c0f5580 msgr2=0x7fcb8c060510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:15.292 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.296+0000 7fcb8a7fc700 1 --2- 172.21.15.139:0/1567479898 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb8c0f5580 0x7fcb8c060510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:15.292 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.296+0000 7fcb8a7fc700 1 -- 172.21.15.139:0/1567479898 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c005040 con 0x7fcb8c0f6820 2021-11-20T06:07:15.292 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.297+0000 7fcb8a7fc700 1 --2- 172.21.15.139:0/1567479898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb8c0f6820 0x7fcb8c10bca0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fcb80002700 tx=0x7fcb80009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:15.292 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.297+0000 7fcb737fe700 1 -- 172.21.15.139:0/1567479898 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb80009320 con 0x7fcb8c0f6820 2021-11-20T06:07:15.293 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.297+0000 7fcb9163c700 1 -- 172.21.15.139:0/1567479898 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8c10c530 con 0x7fcb8c0f6820 2021-11-20T06:07:15.293 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.297+0000 7fcb737fe700 1 -- 172.21.15.139:0/1567479898 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb8000f040 con 0x7fcb8c0f6820 2021-11-20T06:07:15.293 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.297+0000 7fcb737fe700 1 -- 172.21.15.139:0/1567479898 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb800147e0 con 0x7fcb8c0f6820 2021-11-20T06:07:15.293 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.297+0000 7fcb9163c700 1 -- 172.21.15.139:0/1567479898 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8c10caf0 con 0x7fcb8c0f6820 2021-11-20T06:07:15.294 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.298+0000 7fcb9163c700 1 -- 172.21.15.139:0/1567479898 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb8c027a00 con 0x7fcb8c0f6820 2021-11-20T06:07:15.294 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.298+0000 7fcb737fe700 1 -- 172.21.15.139:0/1567479898 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcb80014a50 con 0x7fcb8c0f6820 2021-11-20T06:07:15.296 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.299+0000 7fcb737fe700 1 --2- 172.21.15.139:0/1567479898 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fcb7405c2a0 0x7fcb7405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:15.296 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.299+0000 7fcb737fe700 1 -- 172.21.15.139:0/1567479898 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fcb80076960 con 0x7fcb8c0f6820 2021-11-20T06:07:15.296 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.299+0000 7fcb8affd700 1 --2- 172.21.15.139:0/1567479898 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fcb7405c2a0 0x7fcb7405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:15.296 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.299+0000 7fcb8affd700 1 --2- 172.21.15.139:0/1567479898 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fcb7405c2a0 0x7fcb7405e760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fcb7c009f50 tx=0x7fcb7c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:15.297 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.305+0000 7fcb737fe700 1 -- 172.21.15.139:0/1567479898 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb800491c0 con 0x7fcb8c0f6820 2021-11-20T06:07:15.478 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.493+0000 7fcb9163c700 1 -- 172.21.15.139:0/1567479898 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb8c0fdd10 con 0x7fcb8c0f6820 2021-11-20T06:07:15.479 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.493+0000 7fcb737fe700 1 -- 172.21.15.139:0/1567479898 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2188 (secure 0 0 0) 0x7fcb8004c7e0 con 0x7fcb8c0f6820 2021-11-20T06:07:15.479 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:15.480 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":69,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:15.302341+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24327":{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":69,"state":"up:replay","state_seq":3,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:15.481 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.495+0000 7fcb9163c700 1 -- 172.21.15.139:0/1567479898 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fcb7405c2a0 msgr2=0x7fcb7405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:15.481 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.495+0000 7fcb9163c700 1 --2- 172.21.15.139:0/1567479898 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fcb7405c2a0 0x7fcb7405e760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fcb7c009f50 tx=0x7fcb7c009b30).stop 2021-11-20T06:07:15.481 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.495+0000 7fcb9163c700 1 -- 172.21.15.139:0/1567479898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb8c0f6820 msgr2=0x7fcb8c10bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:15.482 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.495+0000 7fcb9163c700 1 --2- 172.21.15.139:0/1567479898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb8c0f6820 0x7fcb8c10bca0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fcb80002700 tx=0x7fcb80009de0).stop 2021-11-20T06:07:15.482 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.495+0000 7fcb9163c700 1 -- 172.21.15.139:0/1567479898 shutdown_connections 2021-11-20T06:07:15.482 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.495+0000 7fcb9163c700 1 --2- 172.21.15.139:0/1567479898 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fcb7405c2a0 0x7fcb7405e760 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:15.482 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.495+0000 7fcb9163c700 1 --2- 172.21.15.139:0/1567479898 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb8c0f5580 0x7fcb8c060510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:15.482 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.495+0000 7fcb9163c700 1 --2- 172.21.15.139:0/1567479898 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcb8c0f6820 0x7fcb8c10bca0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:15.483 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.495+0000 7fcb9163c700 1 -- 172.21.15.139:0/1567479898 >> 172.21.15.139:0/1567479898 conn(0x7fcb8c0f0ce0 msgr2=0x7fcb8c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:15.483 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.495+0000 7fcb9163c700 1 -- 172.21.15.139:0/1567479898 shutdown_connections 2021-11-20T06:07:15.483 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:15.495+0000 7fcb9163c700 1 -- 172.21.15.139:0/1567479898 wait complete. 2021-11-20T06:07:15.483 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 69 2021-11-20T06:07:15.491 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:15 smithi139 conmon[33501]: debug 2021-11-20T06:07:15.297+0000 7f2a21654700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:07:15.491 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:15 smithi139 conmon[33501]: cluster 2021-11-20T06:07:14.268439+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:07:15.493 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:15 smithi139 conmon[33501]: ) 303 : cluster [DBG] pgmap v276: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:07:15.493 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:15 smithi139 conmon[33501]: cluster 2021-11-20T06:07:15.225315+0000 mon.smithi139 (mon.0) 777 : cluster [INF] daemon mds.cephfs.smithi178.pskihy restarted 2021-11-20T06:07:15.494 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:15 smithi139 conmon[33501]: cluster 2021-11-20T06:07:15. 2021-11-20T06:07:15.494 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:15 smithi139 conmon[33501]: 228971+0000 mon.smithi139 (mon.0) 778 2021-11-20T06:07:15.494 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:15 smithi139 conmon[33501]: : cluster [DBG] osdmap e54: 6 total, 6 up, 6 in 2021-11-20T06:07:15.496 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:15.302341+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24327': {'gid': 24327, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 69, 'state': 'up:replay', 'state_seq': 3, 'addr': '172.21.15.178:6825/3308879813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 3308879813}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 3308879813}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:15.496 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24327:up:replay 2021-11-20T06:07:15.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:15 smithi178 conmon[36600]: cluster 2021-11-20T06:07:14.268439+0000 2021-11-20T06:07:15.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:15 smithi178 conmon[36600]: mgr.smithi139.dplyxn (mgr.14162) 303 : cluster [DBG] pgmap v276: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:07:15.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:15 smithi178 conmon[36600]: cluster 2021-11-20T06:07: 2021-11-20T06:07:15.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:15 smithi178 conmon[36600]: 15.225315+0000 mon.smithi139 (mon.0) 777 : cluster [INF] daemon mds.cephfs.smithi178.pskihy restarted 2021-11-20T06:07:15.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:15 smithi178 conmon[36600]: cluster 2021- 2021-11-20T06:07:15.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:15 smithi178 conmon[36600]: 11-20T06:07:15.228971+0000 2021-11-20T06:07:15.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:15 smithi178 conmon[36600]: mon.smithi139 (mon.0) 778 : cluster 2021-11-20T06:07:15.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:15 smithi178 conmon[36600]: [DBG] osdmap e54: 6 total, 6 up, 6 in 2021-11-20T06:07:16.497 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:16.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.638+0000 7f8c10831700 1 -- 172.21.15.139:0/503745775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c0c0f6680 msgr2=0x7f8c0c0f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:16.626 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.638+0000 7f8c10831700 1 --2- 172.21.15.139:0/503745775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c0c0f6680 0x7f8c0c0f6aa0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f8bfc004660 tx=0x7f8bfc009b30).stop 2021-11-20T06:07:16.626 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.638+0000 7f8c10831700 1 -- 172.21.15.139:0/503745775 shutdown_connections 2021-11-20T06:07:16.626 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.638+0000 7f8c10831700 1 --2- 172.21.15.139:0/503745775 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c0c0f7920 0x7f8c0c0f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:16.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.638+0000 7f8c10831700 1 --2- 172.21.15.139:0/503745775 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c0c0f6680 0x7f8c0c0f6aa0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:16.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.638+0000 7f8c10831700 1 -- 172.21.15.139:0/503745775 >> 172.21.15.139:0/503745775 conn(0x7f8c0c0f1de0 msgr2=0x7f8c0c0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:16.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.638+0000 7f8c10831700 1 -- 172.21.15.139:0/503745775 shutdown_connections 2021-11-20T06:07:16.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.638+0000 7f8c10831700 1 -- 172.21.15.139:0/503745775 wait complete. 2021-11-20T06:07:16.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.638+0000 7f8c10831700 1 Processor -- start 2021-11-20T06:07:16.628 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.638+0000 7f8c10831700 1 -- start start 2021-11-20T06:07:16.628 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.638+0000 7f8c10831700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c0c0f6680 0x7f8c0c106e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:16.628 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.638+0000 7f8c10831700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c0c0f7920 0x7f8c0c107310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:16.628 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.638+0000 7f8c10831700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c1078c0 con 0x7f8c0c0f6680 2021-11-20T06:07:16.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.639+0000 7f8c10831700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c107a00 con 0x7f8c0c0f7920 2021-11-20T06:07:16.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.639+0000 7f8c0a59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c0c0f6680 0x7f8c0c106e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:16.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.639+0000 7f8c0a59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c0c0f6680 0x7f8c0c106e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58688/0 (socket says 172.21.15.139:58688) 2021-11-20T06:07:16.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.639+0000 7f8c0a59c700 1 -- 172.21.15.139:0/3866313510 learned_addr learned my addr 172.21.15.139:0/3866313510 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:16.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.639+0000 7f8c03fff700 1 --2- 172.21.15.139:0/3866313510 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c0c0f7920 0x7f8c0c107310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:16.630 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.639+0000 7f8c0a59c700 1 -- 172.21.15.139:0/3866313510 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c0c0f7920 msgr2=0x7f8c0c107310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:16.630 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.639+0000 7f8c0a59c700 1 --2- 172.21.15.139:0/3866313510 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c0c0f7920 0x7f8c0c107310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:16.630 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.639+0000 7f8c0a59c700 1 -- 172.21.15.139:0/3866313510 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc005040 con 0x7f8c0c0f6680 2021-11-20T06:07:16.630 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.639+0000 7f8c0a59c700 1 --2- 172.21.15.139:0/3866313510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c0c0f6680 0x7f8c0c106e20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f8bfc004e30 tx=0x7f8bfc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:16.631 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.639+0000 7f8c02ffd700 1 -- 172.21.15.139:0/3866313510 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bfc018070 con 0x7f8c0c0f6680 2021-11-20T06:07:16.631 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.639+0000 7f8c02ffd700 1 -- 172.21.15.139:0/3866313510 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8bfc013410 con 0x7f8c0c0f6680 2021-11-20T06:07:16.631 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.639+0000 7f8c10831700 1 -- 172.21.15.139:0/3866313510 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0c10c440 con 0x7f8c0c0f6680 2021-11-20T06:07:16.631 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.639+0000 7f8c02ffd700 1 -- 172.21.15.139:0/3866313510 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bfc01de20 con 0x7f8c0c0f6680 2021-11-20T06:07:16.631 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.639+0000 7f8c10831700 1 -- 172.21.15.139:0/3866313510 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0c10ca00 con 0x7f8c0c0f6680 2021-11-20T06:07:16.632 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.640+0000 7f8c10831700 1 -- 172.21.15.139:0/3866313510 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0c027a00 con 0x7f8c0c0f6680 2021-11-20T06:07:16.632 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.640+0000 7f8c02ffd700 1 -- 172.21.15.139:0/3866313510 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8bfc01d420 con 0x7f8c0c0f6680 2021-11-20T06:07:16.633 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.641+0000 7f8c02ffd700 1 --2- 172.21.15.139:0/3866313510 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8bf805c2a0 0x7f8bf805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:16.633 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.641+0000 7f8c02ffd700 1 -- 172.21.15.139:0/3866313510 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8bfc077f50 con 0x7f8c0c0f6680 2021-11-20T06:07:16.633 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.641+0000 7f8c03fff700 1 --2- 172.21.15.139:0/3866313510 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8bf805c2a0 0x7f8bf805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:16.634 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.641+0000 7f8c03fff700 1 --2- 172.21.15.139:0/3866313510 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8bf805c2a0 0x7f8bf805e760 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f8bf4004ce0 tx=0x7f8bf400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:16.634 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.645+0000 7f8c02ffd700 1 -- 172.21.15.139:0/3866313510 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8bfc0a42b0 con 0x7f8c0c0f6680 2021-11-20T06:07:16.699 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: cluster 2021-11-20T06:07:15.298717+0000 mon.smithi139 (mon.0 2021-11-20T06:07:16.699 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: ) 779 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:07:16.700 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: cluster 2021-11-20T06:07:15.302238+0000 mon.smithi139 (mon.0) 780 : cluster [DBG] mds.? [v2:172.21.15.178:6826/2887891857,v1:172.21.15.178:6827/2887891857] up:boot 2021-11-20T06:07:16.700 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: cluster 2021-11-20T06:07:15.302297+0000 mon.smithi139 (mon.0) 781 : cluster [INF] Standby daemon mds.cephfs.smithi178.blvkbx assigned to filesystem cephfs as rank 0 2021-11-20T06:07:16.700 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: cluster 2021-11-20T06:07:16.700 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: 2021-11-20T06:07: 2021-11-20T06:07:16.701 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: 15.302389+0000 mon.smithi139 (mon.0) 782 : cluster [INF] 2021-11-20T06:07:16.701 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:07:16.701 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: cluster 2021-11-20T06: 2021-11-20T06:07:16.701 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: 07:15.302671+0000 mon.smithi139 (mon.0) 783 : cluster 2021-11-20T06:07:16.702 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T06:07:16.702 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: audit 2021-11-20T 2021-11-20T06:07:16.702 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: 06:07:15.302773+0000 mon.smithi139 (mon.0) 784 2021-11-20T06:07:16.702 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.pskihy"}]: dispatch 2021-11-20T06:07:16.702 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: cluster 2021-11-20T06:07:16.703 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: 2021-11-20T06:07:15.309916+0000 mon.smithi139 2021-11-20T06:07:16.703 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: (mon.0) 785 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:replay} 1 up:standby 2021-11-20T06:07:16.703 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: audit 2021-11-20T06:07:16.703 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: 2021-11-20T06:07:15 2021-11-20T06:07:16.704 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: .494885+0000 mon.smithi139 (mon.0) 786 : audit [DBG] 2021-11-20T06:07:16.704 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:16 smithi178 conmon[36600]: from='client.? 172.21.15.139:0/1567479898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:16.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: cluster 2021-11-20T06:07:15.298717+0000 mon.smithi139 (mon.0) 779 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:07:16.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: cluster 2021-11-20T06:07:15.302238+0000 mon.smithi139 (mon.0) 780 : cluster [DBG] mds.? [v2:172.21.15.178:6826/2887891857,v1:172.21.15.178:6827/2887891857] up:boot 2021-11-20T06:07:16.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: cluster 2021-11-20T06:07:15.302297+0000 mon.smithi139 (mon.0) 781 : cluster [INF] Standby daemon mds.cephfs.smithi178.blvkbx assigned to filesystem cephfs as rank 0 2021-11-20T06:07:16.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: cluster 2021-11-20T06:07:15.302389+0000 mon.smithi139 2021-11-20T06:07:16.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: (mon.0) 782 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:07:16.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: cluster 2021-11-20 2021-11-20T06:07:16.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: T06:07:15.302671+0000 mon.smithi139 (mon.0) 783 : cluster [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T06:07:16.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: audit 2021-11-20T06 2021-11-20T06:07:16.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: :07:15.302773+0000 mon.smithi139 (mon.0) 784 : audit [DBG] 2021-11-20T06:07:16.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.pskihy"}]: dispatch 2021-11-20T06:07:16.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: cluster 2021-11-20T06 2021-11-20T06:07:16.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: :07:15.309916+0000 mon.smithi139 (mon.0) 785 : cluster 2021-11-20T06:07:16.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:replay} 1 up:standby 2021-11-20T06:07:16.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: audit 2021- 2021-11-20T06:07:16.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: 11-20T06:07:15.494885+0000 mon.smithi139 (mon. 2021-11-20T06:07:16.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:16 smithi139 conmon[33501]: 0) 786 : audit [DBG] from='client.? 172.21.15.139:0/1567479898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:16.814 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.828+0000 7f8c10831700 1 -- 172.21.15.139:0/3866313510 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c0c0fee10 con 0x7f8c0c0f6680 2021-11-20T06:07:16.814 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.829+0000 7f8c02ffd700 1 -- 172.21.15.139:0/3866313510 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+2192 (secure 0 0 0) 0x7f8bfc04a7b0 con 0x7f8c0c0f6680 2021-11-20T06:07:16.815 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:16.815 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":70,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:16.307775+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24327":{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":69,"state":"up:reconnect","state_seq":51,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:16.817 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.830+0000 7f8c10831700 1 -- 172.21.15.139:0/3866313510 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8bf805c2a0 msgr2=0x7f8bf805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:16.817 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.830+0000 7f8c10831700 1 --2- 172.21.15.139:0/3866313510 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8bf805c2a0 0x7f8bf805e760 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f8bf4004ce0 tx=0x7f8bf400b040).stop 2021-11-20T06:07:16.817 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.830+0000 7f8c10831700 1 -- 172.21.15.139:0/3866313510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c0c0f6680 msgr2=0x7f8c0c106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:16.817 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.830+0000 7f8c10831700 1 --2- 172.21.15.139:0/3866313510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c0c0f6680 0x7f8c0c106e20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f8bfc004e30 tx=0x7f8bfc009b30).stop 2021-11-20T06:07:16.818 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.831+0000 7f8c10831700 1 -- 172.21.15.139:0/3866313510 shutdown_connections 2021-11-20T06:07:16.818 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.831+0000 7f8c10831700 1 --2- 172.21.15.139:0/3866313510 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8bf805c2a0 0x7f8bf805e760 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:16.818 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.831+0000 7f8c10831700 1 --2- 172.21.15.139:0/3866313510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8c0c0f6680 0x7f8c0c106e20 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:16.819 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.831+0000 7f8c10831700 1 --2- 172.21.15.139:0/3866313510 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8c0c0f7920 0x7f8c0c107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:16.819 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.831+0000 7f8c10831700 1 -- 172.21.15.139:0/3866313510 >> 172.21.15.139:0/3866313510 conn(0x7f8c0c0f1de0 msgr2=0x7f8c0c0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:16.819 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.831+0000 7f8c10831700 1 -- 172.21.15.139:0/3866313510 shutdown_connections 2021-11-20T06:07:16.819 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:16.831+0000 7f8c10831700 1 -- 172.21.15.139:0/3866313510 wait complete. 2021-11-20T06:07:16.820 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 70 2021-11-20T06:07:16.828 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:16.307775+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24327': {'gid': 24327, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 69, 'state': 'up:reconnect', 'state_seq': 51, 'addr': '172.21.15.178:6825/3308879813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 3308879813}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 3308879813}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:16.829 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24327:up:reconnect 2021-11-20T06:07:17.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:17 smithi178 conmon[36600]: cluster 2021-11-20T06:07:16.269035+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:07:17.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:17 smithi178 conmon[36600]: ) 304 : cluster [DBG] pgmap v278: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-20T06:07:17.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:17 smithi178 conmon[36600]: cluster 2021-11-20T06:07:16.314486+0000 mon.smithi139 (mon.0) 787 : cluster [DBG] mds.? [v2:172.21.15.178:6824/3308879813,v1:172.21.15.178:6825/3308879813] up:reconnect 2021-11-20T06:07:17.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:17 smithi178 conmon[36600]: cluster 2021-11-20T06:07:16.314544+0000 mon.smithi139 (mon.0) 2021-11-20T06:07:17.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:17 smithi178 conmon[36600]: 788 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:reconnect} 1 up:standby 2021-11-20T06:07:17.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:17 smithi178 conmon[36600]: cluster 2021-11-20T06:07:16.334513+0000 mds.cephfs.smithi178.blvkbx (mds. 2021-11-20T06:07:17.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:17 smithi178 conmon[36600]: 0) 1 : cluster [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0209996 2021-11-20T06:07:17.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:17 smithi178 conmon[36600]: audit 2021-11-20T06: 2021-11-20T06:07:17.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:17 smithi178 conmon[36600]: 07:16.830686+0000 mon.smithi139 (mon.0) 2021-11-20T06:07:17.444 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:17 smithi178 conmon[36600]: 789 : audit [DBG] from='client.? 172.21.15.139:0/3866313510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:17.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:17 smithi139 conmon[33501]: cluster 2021-11-20T06:07:16.269035+0000 mgr.smithi139.dplyxn (mgr.14162) 304 2021-11-20T06:07:17.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:17 smithi139 conmon[33501]: : cluster [DBG] pgmap v278: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-20T06:07:17.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:17 smithi139 conmon[33501]: cluster 2021-11-20T06:07:16.314486+0000 mon.smithi139 (mon.0) 787 : cluster [DBG] mds.? [v2:172.21.15.178:6824/3308879813,v1:172.21.15.178:6825/3308879813] up:reconnect 2021-11-20T06:07:17.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:17 smithi139 conmon[33501]: cluster 2021 2021-11-20T06:07:17.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:17 smithi139 conmon[33501]: -11-20T06:07:16.314544+0000 mon.smithi139 (mon.0) 788 : cluster 2021-11-20T06:07:17.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:17 smithi139 conmon[33501]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:reconnect} 1 up:standby 2021-11-20T06:07:17.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:17 smithi139 conmon[33501]: cluster 2021-11-20T06:07:17.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:17 smithi139 conmon[33501]: 2021-11-20T06:07 2021-11-20T06:07:17.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:17 smithi139 conmon[33501]: :16.334513+0000 mds.cephfs.smithi178.blvkbx (mds.0) 2021-11-20T06:07:17.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:17 smithi139 conmon[33501]: 1 : cluster [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0209996 2021-11-20T06:07:17.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:17 smithi139 conmon[33501]: audit 2021 2021-11-20T06:07:17.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:17 smithi139 conmon[33501]: -11-20T06:07: 2021-11-20T06:07:17.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:17 smithi139 conmon[33501]: 16.830686+0000 mon.smithi139 (mon.0) 789 : 2021-11-20T06:07:17.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:17 smithi139 conmon[33501]: audit [DBG] from='client.? 172.21.15.139:0/3866313510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:17.830 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:17.960 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.972+0000 7f0198454700 1 -- 172.21.15.139:0/1415907373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0190060690 msgr2=0x7f01900f92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:17.961 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.972+0000 7f0198454700 1 --2- 172.21.15.139:0/1415907373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0190060690 0x7f01900f92f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f018c004660 tx=0x7f018c009b30).stop 2021-11-20T06:07:17.961 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.972+0000 7f0198454700 1 -- 172.21.15.139:0/1415907373 shutdown_connections 2021-11-20T06:07:17.961 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.972+0000 7f0198454700 1 --2- 172.21.15.139:0/1415907373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01900f97e0 0x7f01900fbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:17.961 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.972+0000 7f0198454700 1 --2- 172.21.15.139:0/1415907373 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0190060690 0x7f01900f92f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:17.961 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.972+0000 7f0198454700 1 -- 172.21.15.139:0/1415907373 >> 172.21.15.139:0/1415907373 conn(0x7f01900f0cf0 msgr2=0x7f01900f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:17.962 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.972+0000 7f0198454700 1 -- 172.21.15.139:0/1415907373 shutdown_connections 2021-11-20T06:07:17.962 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.972+0000 7f0198454700 1 -- 172.21.15.139:0/1415907373 wait complete. 2021-11-20T06:07:17.962 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.972+0000 7f0198454700 1 Processor -- start 2021-11-20T06:07:17.963 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.972+0000 7f0198454700 1 -- start start 2021-11-20T06:07:17.963 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f0198454700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0190060690 0x7f0190106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:17.963 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f0198454700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01900f97e0 0x7f01901073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:17.963 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f0198454700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0190107970 con 0x7f0190060690 2021-11-20T06:07:17.964 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f0198454700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0190107ab0 con 0x7f01900f97e0 2021-11-20T06:07:17.964 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f01961f0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0190060690 0x7f0190106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:17.964 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f01961f0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0190060690 0x7f0190106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58698/0 (socket says 172.21.15.139:58698) 2021-11-20T06:07:17.964 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f01961f0700 1 -- 172.21.15.139:0/992845155 learned_addr learned my addr 172.21.15.139:0/992845155 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:17.965 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f01959ef700 1 --2- 172.21.15.139:0/992845155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01900f97e0 0x7f01901073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:17.965 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f01961f0700 1 -- 172.21.15.139:0/992845155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01900f97e0 msgr2=0x7f01901073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:17.965 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f01961f0700 1 --2- 172.21.15.139:0/992845155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01900f97e0 0x7f01901073c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:17.965 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f01961f0700 1 -- 172.21.15.139:0/992845155 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018c005040 con 0x7f0190060690 2021-11-20T06:07:17.966 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f01959ef700 1 --2- 172.21.15.139:0/992845155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01900f97e0 0x7f01901073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:07:17.966 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f01961f0700 1 --2- 172.21.15.139:0/992845155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0190060690 0x7f0190106ed0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f018c009830 tx=0x7f018c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:17.966 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f0186ffd700 1 -- 172.21.15.139:0/992845155 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f018c018070 con 0x7f0190060690 2021-11-20T06:07:17.967 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f0186ffd700 1 -- 172.21.15.139:0/992845155 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f018c013410 con 0x7f0190060690 2021-11-20T06:07:17.967 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f0198454700 1 -- 172.21.15.139:0/992845155 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019010c4f0 con 0x7f0190060690 2021-11-20T06:07:17.968 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f0186ffd700 1 -- 172.21.15.139:0/992845155 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f018c01c620 con 0x7f0190060690 2021-11-20T06:07:17.968 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.973+0000 7f0198454700 1 -- 172.21.15.139:0/992845155 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f019010cab0 con 0x7f0190060690 2021-11-20T06:07:17.969 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.974+0000 7f0198454700 1 -- 172.21.15.139:0/992845155 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0190027a00 con 0x7f0190060690 2021-11-20T06:07:17.969 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.975+0000 7f0186ffd700 1 -- 172.21.15.139:0/992845155 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f018c01d960 con 0x7f0190060690 2021-11-20T06:07:17.969 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.975+0000 7f0186ffd700 1 --2- 172.21.15.139:0/992845155 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f017c05c2a0 0x7f017c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:17.969 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.975+0000 7f0186ffd700 1 -- 172.21.15.139:0/992845155 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f018c078290 con 0x7f0190060690 2021-11-20T06:07:17.970 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.975+0000 7f01959ef700 1 --2- 172.21.15.139:0/992845155 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f017c05c2a0 0x7f017c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:17.970 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.975+0000 7f01959ef700 1 --2- 172.21.15.139:0/992845155 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f017c05c2a0 0x7f017c05e760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f01800041d0 tx=0x7f0180009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:17.971 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:17.979+0000 7f0186ffd700 1 -- 172.21.15.139:0/992845155 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f018c04aa90 con 0x7f0190060690 2021-11-20T06:07:18.151 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:18.165+0000 7f0198454700 1 -- 172.21.15.139:0/992845155 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01900fee30 con 0x7f0190060690 2021-11-20T06:07:18.152 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:18.166+0000 7f0186ffd700 1 -- 172.21.15.139:0/992845155 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2189 (secure 0 0 0) 0x7f018c04e0b0 con 0x7f0190060690 2021-11-20T06:07:18.152 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:18.152 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:17.312170+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24327":{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":52,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:18.154 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:18.167+0000 7f0198454700 1 -- 172.21.15.139:0/992845155 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f017c05c2a0 msgr2=0x7f017c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:18.154 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:18.167+0000 7f0198454700 1 --2- 172.21.15.139:0/992845155 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f017c05c2a0 0x7f017c05e760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f01800041d0 tx=0x7f0180009040).stop 2021-11-20T06:07:18.154 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:18.167+0000 7f0198454700 1 -- 172.21.15.139:0/992845155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0190060690 msgr2=0x7f0190106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:18.154 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:18.167+0000 7f0198454700 1 --2- 172.21.15.139:0/992845155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0190060690 0x7f0190106ed0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f018c009830 tx=0x7f018c009b30).stop 2021-11-20T06:07:18.155 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:18.167+0000 7f0198454700 1 -- 172.21.15.139:0/992845155 shutdown_connections 2021-11-20T06:07:18.155 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:18.167+0000 7f0198454700 1 --2- 172.21.15.139:0/992845155 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f017c05c2a0 0x7f017c05e760 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:18.155 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:18.167+0000 7f0198454700 1 --2- 172.21.15.139:0/992845155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0190060690 0x7f0190106ed0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:18.155 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:18.167+0000 7f0198454700 1 --2- 172.21.15.139:0/992845155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01900f97e0 0x7f01901073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:18.155 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:18.167+0000 7f0198454700 1 -- 172.21.15.139:0/992845155 >> 172.21.15.139:0/992845155 conn(0x7f01900f0cf0 msgr2=0x7f01900f30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:18.156 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:18.168+0000 7f0198454700 1 -- 172.21.15.139:0/992845155 shutdown_connections 2021-11-20T06:07:18.156 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:18.168+0000 7f0198454700 1 -- 172.21.15.139:0/992845155 wait complete. 2021-11-20T06:07:18.156 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 71 2021-11-20T06:07:18.165 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:17.312170+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24327': {'gid': 24327, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 52, 'addr': '172.21.15.178:6825/3308879813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 3308879813}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 3308879813}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:18.165 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24327:up:rejoin 2021-11-20T06:07:18.501 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:18 smithi139 conmon[33501]: cluster 2021-11-20T 2021-11-20T06:07:18.502 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:18 smithi139 conmon[33501]: 06:07:17.314800+0000 mon.smithi139 (mon.0) 790 : cluster [DBG] mds.? [v2:172.21.15.178:6824/3308879813,v1:172.21.15.178:6825/3308879813] up:rejoin 2021-11-20T06:07:18.502 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:18 smithi139 conmon[33501]: cluster 2021-11-20T06:07:17.314865+0000 2021-11-20T06:07:18.502 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:18 smithi139 conmon[33501]: mon.smithi139 (mon.0) 791 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:rejoin} 1 up:standby 2021-11-20T06:07:18.503 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:18 smithi139 conmon[33501]: audit 2021-11-20T06:07:18.503 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:18 smithi139 conmon[33501]: 2021-11-20T06:07: 2021-11-20T06:07:18.503 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:18 smithi139 conmon[33501]: 18.167806+0000 mon.smithi139 (mon.0) 792 : audit 2021-11-20T06:07:18.504 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:18 smithi139 conmon[33501]: [DBG] from='client.? 172.21.15.139:0/992845155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:18.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:18 smithi178 conmon[36600]: cluster 2021-11-20T06:07:17.314800+0000 mon.smithi139 (mon.0) 790 : cluster [DBG] mds.? [v2:172.21.15.178:6824/3308879813,v1:172.21.15.178:6825/3308879813] up:rejoin 2021-11-20T06:07:18.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:18 smithi178 conmon[36600]: cluster 2021-11-20T06:07:17.314865+0000 mon.smithi139 ( 2021-11-20T06:07:18.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:18 smithi178 conmon[36600]: mon.0) 791 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:rejoin} 1 up:standby 2021-11-20T06:07:18.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:18 smithi178 conmon[36600]: audit 2021-11-20T06:07:18.167806+0000 mon.smithi139 (mon.0) 792 : audit [DBG] from='client.? 172.21.15.139:0/992845155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:19.167 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:19.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.300+0000 7f53c2d38700 1 -- 172.21.15.139:0/2992993491 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53bc0f7570 msgr2=0x7f53bc0f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:19.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.300+0000 7f53c2d38700 1 --2- 172.21.15.139:0/2992993491 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53bc0f7570 0x7f53bc0f5990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f53a8004660 tx=0x7f53a8009b30).stop 2021-11-20T06:07:19.289 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.300+0000 7f53c2d38700 1 -- 172.21.15.139:0/2992993491 shutdown_connections 2021-11-20T06:07:19.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.300+0000 7f53c2d38700 1 --2- 172.21.15.139:0/2992993491 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53bc0f7570 0x7f53bc0f5990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:19.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.300+0000 7f53c2d38700 1 --2- 172.21.15.139:0/2992993491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53bc0f6bd0 0x7f53bc0f6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:19.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.300+0000 7f53c2d38700 1 -- 172.21.15.139:0/2992993491 >> 172.21.15.139:0/2992993491 conn(0x7f53bc0f04a0 msgr2=0x7f53bc0f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:19.290 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.300+0000 7f53c2d38700 1 -- 172.21.15.139:0/2992993491 shutdown_connections 2021-11-20T06:07:19.291 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.300+0000 7f53c2d38700 1 -- 172.21.15.139:0/2992993491 wait complete. 2021-11-20T06:07:19.291 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.301+0000 7f53c2d38700 1 Processor -- start 2021-11-20T06:07:19.291 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.301+0000 7f53c2d38700 1 -- start start 2021-11-20T06:07:19.291 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.301+0000 7f53c2d38700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53bc0f6bd0 0x7f53bc0f5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:19.292 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.301+0000 7f53c2d38700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53bc0f7570 0x7f53bc10bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:19.292 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.301+0000 7f53c2d38700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc10c130 con 0x7f53bc0f6bd0 2021-11-20T06:07:19.292 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.301+0000 7f53c2d38700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc10c270 con 0x7f53bc0f7570 2021-11-20T06:07:19.292 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.301+0000 7f53c0ad4700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53bc0f6bd0 0x7f53bc0f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:19.293 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.301+0000 7f53c0ad4700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53bc0f6bd0 0x7f53bc0f5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58714/0 (socket says 172.21.15.139:58714) 2021-11-20T06:07:19.293 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.301+0000 7f53c0ad4700 1 -- 172.21.15.139:0/1253017276 learned_addr learned my addr 172.21.15.139:0/1253017276 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:19.293 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.302+0000 7f53bbfff700 1 --2- 172.21.15.139:0/1253017276 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53bc0f7570 0x7f53bc10bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:19.293 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.302+0000 7f53c0ad4700 1 -- 172.21.15.139:0/1253017276 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53bc0f7570 msgr2=0x7f53bc10bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:19.294 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.302+0000 7f53c0ad4700 1 --2- 172.21.15.139:0/1253017276 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53bc0f7570 0x7f53bc10bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:19.294 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.302+0000 7f53c0ad4700 1 -- 172.21.15.139:0/1253017276 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a8005040 con 0x7f53bc0f6bd0 2021-11-20T06:07:19.294 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.302+0000 7f53bbfff700 1 --2- 172.21.15.139:0/1253017276 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53bc0f7570 0x7f53bc10bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:07:19.294 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.302+0000 7f53c0ad4700 1 --2- 172.21.15.139:0/1253017276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53bc0f6bd0 0x7f53bc0f5370 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f53b0002700 tx=0x7f53b0009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:19.295 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.302+0000 7f53b97fa700 1 -- 172.21.15.139:0/1253017276 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53b0009320 con 0x7f53bc0f6bd0 2021-11-20T06:07:19.295 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.302+0000 7f53c2d38700 1 -- 172.21.15.139:0/1253017276 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53bc10c500 con 0x7f53bc0f6bd0 2021-11-20T06:07:19.295 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.302+0000 7f53b97fa700 1 -- 172.21.15.139:0/1253017276 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53b000f040 con 0x7f53bc0f6bd0 2021-11-20T06:07:19.295 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.302+0000 7f53b97fa700 1 -- 172.21.15.139:0/1253017276 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53b0014900 con 0x7f53bc0f6bd0 2021-11-20T06:07:19.296 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.302+0000 7f53c2d38700 1 -- 172.21.15.139:0/1253017276 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc10cac0 con 0x7f53bc0f6bd0 2021-11-20T06:07:19.296 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.303+0000 7f53c2d38700 1 -- 172.21.15.139:0/1253017276 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53bc027a00 con 0x7f53bc0f6bd0 2021-11-20T06:07:19.296 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.303+0000 7f53b97fa700 1 -- 172.21.15.139:0/1253017276 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f53b0009480 con 0x7f53bc0f6bd0 2021-11-20T06:07:19.296 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.303+0000 7f53b97fa700 1 --2- 172.21.15.139:0/1253017276 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f53ac05c2f0 0x7f53ac05e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:19.296 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.303+0000 7f53b97fa700 1 -- 172.21.15.139:0/1253017276 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f53b0076720 con 0x7f53bc0f6bd0 2021-11-20T06:07:19.297 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.304+0000 7f53bbfff700 1 --2- 172.21.15.139:0/1253017276 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f53ac05c2f0 0x7f53ac05e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:19.298 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.304+0000 7f53bbfff700 1 --2- 172.21.15.139:0/1253017276 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f53ac05c2f0 0x7f53ac05e7b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f53a8009d60 tx=0x7f53a8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:19.298 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.307+0000 7f53b97fa700 1 -- 172.21.15.139:0/1253017276 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53b0048ec0 con 0x7f53bc0f6bd0 2021-11-20T06:07:19.479 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.494+0000 7f53c2d38700 1 -- 172.21.15.139:0/1253017276 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53bc0fdd60 con 0x7f53bc0f6bd0 2021-11-20T06:07:19.480 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.494+0000 7f53b97fa700 1 -- 172.21.15.139:0/1253017276 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f53b004c4e0 con 0x7f53bc0f6bd0 2021-11-20T06:07:19.480 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:19.481 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:17.312170+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24327":{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":52,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:19.482 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.496+0000 7f53c2d38700 1 -- 172.21.15.139:0/1253017276 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f53ac05c2f0 msgr2=0x7f53ac05e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:19.482 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.496+0000 7f53c2d38700 1 --2- 172.21.15.139:0/1253017276 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f53ac05c2f0 0x7f53ac05e7b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f53a8009d60 tx=0x7f53a8009b30).stop 2021-11-20T06:07:19.482 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.496+0000 7f53c2d38700 1 -- 172.21.15.139:0/1253017276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53bc0f6bd0 msgr2=0x7f53bc0f5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:19.482 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.496+0000 7f53c2d38700 1 --2- 172.21.15.139:0/1253017276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53bc0f6bd0 0x7f53bc0f5370 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f53b0002700 tx=0x7f53b0009d80).stop 2021-11-20T06:07:19.483 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.496+0000 7f53c2d38700 1 -- 172.21.15.139:0/1253017276 shutdown_connections 2021-11-20T06:07:19.483 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.496+0000 7f53c2d38700 1 --2- 172.21.15.139:0/1253017276 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f53ac05c2f0 0x7f53ac05e7b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:19.483 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.496+0000 7f53c2d38700 1 --2- 172.21.15.139:0/1253017276 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53bc0f6bd0 0x7f53bc0f5370 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:19.483 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.496+0000 7f53c2d38700 1 --2- 172.21.15.139:0/1253017276 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53bc0f7570 0x7f53bc10bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:19.484 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.496+0000 7f53c2d38700 1 -- 172.21.15.139:0/1253017276 >> 172.21.15.139:0/1253017276 conn(0x7f53bc0f04a0 msgr2=0x7f53bc0fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:19.484 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.496+0000 7f53c2d38700 1 -- 172.21.15.139:0/1253017276 shutdown_connections 2021-11-20T06:07:19.484 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:19.496+0000 7f53c2d38700 1 -- 172.21.15.139:0/1253017276 wait complete. 2021-11-20T06:07:19.485 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 72 2021-11-20T06:07:19.493 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:17.312170+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24327': {'gid': 24327, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 52, 'addr': '172.21.15.178:6825/3308879813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 3308879813}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 3308879813}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:19.494 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24327:up:rejoin 2021-11-20T06:07:19.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:19 smithi178 conmon[36600]: cluster 2021-11-20T06:07:18.269524+0000 mgr.smithi139.dplyxn 2021-11-20T06:07:19.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:19 smithi178 conmon[36600]: (mgr.14162) 305 : cluster [DBG] pgmap v279: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-20T06:07:19.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:19 smithi139 conmon[33501]: cluster 2021-11-20T06:07:18.269524+0000 mgr.smithi139.dplyxn 2021-11-20T06:07:19.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:19 smithi139 conmon[33501]: (mgr.14162) 305 : cluster [DBG] pgmap v279: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-20T06:07:20.496 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:20.523 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:20 smithi139 conmon[33501]: cluster 2021-11-20T06:07:19.323998+0000 mon.smithi139 (mon.0) 793 : cluster [DBG] 2021-11-20T06:07:20.523 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:20 smithi139 conmon[33501]: mds.? [v2:172.21.15.178:6826/2887891857,v1:172.21.15.178:6827/2887891857] up:standby 2021-11-20T06:07:20.524 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:20 smithi139 conmon[33501]: cluster 2021-11-20T06:07:19.324048+0000 mon.smithi139 (mon.0) 794 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:rejoin} 1 up:standby 2021-11-20T06:07:20.524 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:20 smithi139 conmon[33501]: audit 2021-11-20T06:07:19.496137+0000 2021-11-20T06:07:20.524 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:20 smithi139 conmon[33501]: mon.smithi139 (mon.0) 795 : audit [DBG] from='client.? 172.21.15.139:0/1253017276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:20.623 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.635+0000 7f0593283700 1 -- 172.21.15.139:0/3046668419 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f058c0f7d90 msgr2=0x7f058c0f81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:20.623 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.635+0000 7f0593283700 1 --2- 172.21.15.139:0/3046668419 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f058c0f7d90 0x7f058c0f81b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f0588004660 tx=0x7f0588009b30).stop 2021-11-20T06:07:20.624 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.635+0000 7f0593283700 1 -- 172.21.15.139:0/3046668419 shutdown_connections 2021-11-20T06:07:20.624 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.635+0000 7f0593283700 1 --2- 172.21.15.139:0/3046668419 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f058c0f4780 0x7f058c0f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:20.624 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.635+0000 7f0593283700 1 --2- 172.21.15.139:0/3046668419 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f058c0f7d90 0x7f058c0f81b0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:20.624 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.635+0000 7f0593283700 1 -- 172.21.15.139:0/3046668419 >> 172.21.15.139:0/3046668419 conn(0x7f058c0f04a0 msgr2=0x7f058c0f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:20.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.635+0000 7f0593283700 1 -- 172.21.15.139:0/3046668419 shutdown_connections 2021-11-20T06:07:20.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.635+0000 7f0593283700 1 -- 172.21.15.139:0/3046668419 wait complete. 2021-11-20T06:07:20.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.636+0000 7f0593283700 1 Processor -- start 2021-11-20T06:07:20.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.636+0000 7f0593283700 1 -- start start 2021-11-20T06:07:20.625 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.636+0000 7f0593283700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f058c0f4780 0x7f058c0f7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:20.626 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.636+0000 7f0593283700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f058c0f7d90 0x7f058c0f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:20.626 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.636+0000 7f0593283700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c0f7900 con 0x7f058c0f4780 2021-11-20T06:07:20.626 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.636+0000 7f0593283700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c0f5ea0 con 0x7f058c0f7d90 2021-11-20T06:07:20.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.636+0000 7f059101f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f058c0f4780 0x7f058c0f7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:20.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.636+0000 7f059101f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f058c0f4780 0x7f058c0f7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58724/0 (socket says 172.21.15.139:58724) 2021-11-20T06:07:20.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.636+0000 7f059101f700 1 -- 172.21.15.139:0/1822605553 learned_addr learned my addr 172.21.15.139:0/1822605553 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:20.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.636+0000 7f059101f700 1 -- 172.21.15.139:0/1822605553 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f058c0f7d90 msgr2=0x7f058c0f59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:20.627 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.636+0000 7f059101f700 1 --2- 172.21.15.139:0/1822605553 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f058c0f7d90 0x7f058c0f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:07:20.628 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.636+0000 7f059101f700 1 -- 172.21.15.139:0/1822605553 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0588005040 con 0x7f058c0f4780 2021-11-20T06:07:20.628 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.636+0000 7f059101f700 1 --2- 172.21.15.139:0/1822605553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f058c0f4780 0x7f058c0f7350 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f0588009f50 tx=0x7f0588009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:20.628 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.637+0000 7f0581ffb700 1 -- 172.21.15.139:0/1822605553 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0588018070 con 0x7f058c0f4780 2021-11-20T06:07:20.628 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.637+0000 7f0581ffb700 1 -- 172.21.15.139:0/1822605553 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0588013410 con 0x7f058c0f4780 2021-11-20T06:07:20.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.637+0000 7f0581ffb700 1 -- 172.21.15.139:0/1822605553 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f058801c620 con 0x7f058c0f4780 2021-11-20T06:07:20.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.637+0000 7f0593283700 1 -- 172.21.15.139:0/1822605553 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058c0f60c0 con 0x7f058c0f4780 2021-11-20T06:07:20.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.637+0000 7f0593283700 1 -- 172.21.15.139:0/1822605553 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058c0f6620 con 0x7f058c0f4780 2021-11-20T06:07:20.629 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.638+0000 7f0593283700 1 -- 172.21.15.139:0/1822605553 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f058c027a00 con 0x7f058c0f4780 2021-11-20T06:07:20.630 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.638+0000 7f0581ffb700 1 -- 172.21.15.139:0/1822605553 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f058801d420 con 0x7f058c0f4780 2021-11-20T06:07:20.631 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.638+0000 7f0581ffb700 1 --2- 172.21.15.139:0/1822605553 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f057805c2a0 0x7f057805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:20.631 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.638+0000 7f0581ffb700 1 -- 172.21.15.139:0/1822605553 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0588078110 con 0x7f058c0f4780 2021-11-20T06:07:20.631 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.639+0000 7f059081e700 1 --2- 172.21.15.139:0/1822605553 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f057805c2a0 0x7f057805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:20.631 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.639+0000 7f059081e700 1 --2- 172.21.15.139:0/1822605553 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f057805c2a0 0x7f057805e760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f058c0f7b60 tx=0x7f057c007a60).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:20.632 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.642+0000 7f0581ffb700 1 -- 172.21.15.139:0/1822605553 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f058804a9f0 con 0x7f058c0f4780 2021-11-20T06:07:20.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:20 smithi178 conmon[36600]: cluster 2021-11-20T06:07:19.323998+0000 mon.smithi139 (mon.0) 793 : cluster [DBG] mds.? [v2:172.21.15.178:6826/2887891857,v1:172.21.15.178:6827/2887891857] up:standby 2021-11-20T06:07:20.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:20 smithi178 conmon[36600]: cluster 2021-11-20T06:07:19.324048+0000 mon.smithi139 (mon.0) 794 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:rejoin} 1 up:standby 2021-11-20T06:07:20.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:20 smithi178 conmon[36600]: audit 2021-11-20T06:07:19.496137+0000 mon.smithi139 (mon.0) 795 : audit 2021-11-20T06:07:20.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:20 smithi178 conmon[36600]: [DBG] from='client.? 172.21.15.139:0/1253017276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:20.812 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.827+0000 7f0593283700 1 -- 172.21.15.139:0/1822605553 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f058c0fef40 con 0x7f058c0f4780 2021-11-20T06:07:20.813 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.827+0000 7f0581ffb700 1 -- 172.21.15.139:0/1822605553 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f058804e010 con 0x7f058c0f4780 2021-11-20T06:07:20.813 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:20.814 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:17.312170+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24327":{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":52,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:20.815 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.829+0000 7f0593283700 1 -- 172.21.15.139:0/1822605553 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f057805c2a0 msgr2=0x7f057805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:20.815 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.829+0000 7f0593283700 1 --2- 172.21.15.139:0/1822605553 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f057805c2a0 0x7f057805e760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f058c0f7b60 tx=0x7f057c007a60).stop 2021-11-20T06:07:20.815 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.829+0000 7f0593283700 1 -- 172.21.15.139:0/1822605553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f058c0f4780 msgr2=0x7f058c0f7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:20.816 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.829+0000 7f0593283700 1 --2- 172.21.15.139:0/1822605553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f058c0f4780 0x7f058c0f7350 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f0588009f50 tx=0x7f0588009b30).stop 2021-11-20T06:07:20.816 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.829+0000 7f0593283700 1 -- 172.21.15.139:0/1822605553 shutdown_connections 2021-11-20T06:07:20.816 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.829+0000 7f0593283700 1 --2- 172.21.15.139:0/1822605553 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f057805c2a0 0x7f057805e760 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:20.816 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.829+0000 7f0593283700 1 --2- 172.21.15.139:0/1822605553 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f058c0f4780 0x7f058c0f7350 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:20.817 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.829+0000 7f0593283700 1 --2- 172.21.15.139:0/1822605553 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f058c0f7d90 0x7f058c0f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:07:20.817 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.829+0000 7f0593283700 1 -- 172.21.15.139:0/1822605553 >> 172.21.15.139:0/1822605553 conn(0x7f058c0f04a0 msgr2=0x7f058c0fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:20.819 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.829+0000 7f0593283700 1 -- 172.21.15.139:0/1822605553 shutdown_connections 2021-11-20T06:07:20.819 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:20.829+0000 7f0593283700 1 -- 172.21.15.139:0/1822605553 wait complete. 2021-11-20T06:07:20.819 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 72 2021-11-20T06:07:20.826 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:17.312170+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24327': {'gid': 24327, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 52, 'addr': '172.21.15.178:6825/3308879813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 3308879813}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 3308879813}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:20.827 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24327:up:rejoin 2021-11-20T06:07:21.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:21 smithi178 conmon[36600]: cluster 2021-11-20T06:07:20.269952+0000 mgr.smithi139.dplyxn (mgr.14162) 306 : cluster [DBG] pgmap v280: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:21.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:21 smithi178 conmon[36600]: audit 2021-11-20T06:07: 2021-11-20T06:07:21.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:21 smithi178 conmon[36600]: 20.828953+0000 mon.smithi139 (mon.0) 796 : audit [DBG] from='client.? 172.21.15.139:0/1822605553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:21.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:21 smithi139 conmon[33501]: cluster 2021-11-20T06:07:20.269952+0000 mgr.smithi139.dplyxn (mgr.14162) 306 : 2021-11-20T06:07:21.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:21 smithi139 conmon[33501]: cluster [DBG] pgmap v280: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:21.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:21 smithi139 conmon[33501]: audit 2021-11-20T06:07:20.828953+0000 mon.smithi139 (mon.0) 796 : audit [DBG] from='client.? 172.21.15.139:0/1822605553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:21.828 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:21.956 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.968+0000 7f51513d7700 1 -- 172.21.15.139:0/2466859030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f514c0f6680 msgr2=0x7f514c0f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:21.956 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.968+0000 7f51513d7700 1 --2- 172.21.15.139:0/2466859030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f514c0f6680 0x7f514c0f6aa0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f513c004660 tx=0x7f513c009b30).stop 2021-11-20T06:07:21.956 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.968+0000 7f51513d7700 1 -- 172.21.15.139:0/2466859030 shutdown_connections 2021-11-20T06:07:21.957 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.968+0000 7f51513d7700 1 --2- 172.21.15.139:0/2466859030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f514c0f7920 0x7f514c0f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:21.957 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.968+0000 7f51513d7700 1 --2- 172.21.15.139:0/2466859030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f514c0f6680 0x7f514c0f6aa0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:21.957 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.968+0000 7f51513d7700 1 -- 172.21.15.139:0/2466859030 >> 172.21.15.139:0/2466859030 conn(0x7f514c0f1de0 msgr2=0x7f514c0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:21.957 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.968+0000 7f51513d7700 1 -- 172.21.15.139:0/2466859030 shutdown_connections 2021-11-20T06:07:21.958 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.968+0000 7f51513d7700 1 -- 172.21.15.139:0/2466859030 wait complete. 2021-11-20T06:07:21.958 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.969+0000 7f51513d7700 1 Processor -- start 2021-11-20T06:07:21.958 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.969+0000 7f51513d7700 1 -- start start 2021-11-20T06:07:21.958 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.969+0000 7f51513d7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f514c0f6680 0x7f514c06b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:21.959 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.969+0000 7f51513d7700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f514c0f7920 0x7f514c069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:21.959 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.969+0000 7f51513d7700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c06bd70 con 0x7f514c0f6680 2021-11-20T06:07:21.959 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.969+0000 7f51513d7700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c06a310 con 0x7f514c0f7920 2021-11-20T06:07:21.959 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.969+0000 7f514affd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f514c0f6680 0x7f514c06b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:21.959 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.969+0000 7f514affd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f514c0f6680 0x7f514c06b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58734/0 (socket says 172.21.15.139:58734) 2021-11-20T06:07:21.960 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.969+0000 7f514affd700 1 -- 172.21.15.139:0/3423953575 learned_addr learned my addr 172.21.15.139:0/3423953575 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:21.960 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.969+0000 7f514affd700 1 -- 172.21.15.139:0/3423953575 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f514c0f7920 msgr2=0x7f514c069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:21.960 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.969+0000 7f514a7fc700 1 --2- 172.21.15.139:0/3423953575 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f514c0f7920 0x7f514c069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:21.960 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.969+0000 7f514affd700 1 --2- 172.21.15.139:0/3423953575 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f514c0f7920 0x7f514c069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:21.961 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.969+0000 7f514affd700 1 -- 172.21.15.139:0/3423953575 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c005040 con 0x7f514c0f6680 2021-11-20T06:07:21.961 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.970+0000 7f514affd700 1 --2- 172.21.15.139:0/3423953575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f514c0f6680 0x7f514c06b7c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f513c004e30 tx=0x7f513c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:21.961 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.970+0000 7f51337fe700 1 -- 172.21.15.139:0/3423953575 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f513c018070 con 0x7f514c0f6680 2021-11-20T06:07:21.961 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.970+0000 7f51513d7700 1 -- 172.21.15.139:0/3423953575 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514c06a530 con 0x7f514c0f6680 2021-11-20T06:07:21.961 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.970+0000 7f51337fe700 1 -- 172.21.15.139:0/3423953575 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f513c013410 con 0x7f514c0f6680 2021-11-20T06:07:21.962 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.970+0000 7f51337fe700 1 -- 172.21.15.139:0/3423953575 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f513c01c3c0 con 0x7f514c0f6680 2021-11-20T06:07:21.962 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.970+0000 7f51513d7700 1 -- 172.21.15.139:0/3423953575 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c06aa90 con 0x7f514c0f6680 2021-11-20T06:07:21.962 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.971+0000 7f51513d7700 1 -- 172.21.15.139:0/3423953575 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514c046330 con 0x7f514c0f6680 2021-11-20T06:07:21.964 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.971+0000 7f51337fe700 1 -- 172.21.15.139:0/3423953575 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f513c01d420 con 0x7f514c0f6680 2021-11-20T06:07:21.964 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.971+0000 7f51337fe700 1 --2- 172.21.15.139:0/3423953575 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f513405c2f0 0x7f513405e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:21.964 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.972+0000 7f51337fe700 1 -- 172.21.15.139:0/3423953575 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f513c078080 con 0x7f514c0f6680 2021-11-20T06:07:21.965 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.972+0000 7f514a7fc700 1 --2- 172.21.15.139:0/3423953575 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f513405c2f0 0x7f513405e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:21.965 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.972+0000 7f514a7fc700 1 --2- 172.21.15.139:0/3423953575 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f513405c2f0 0x7f513405e7b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f5140004a70 tx=0x7f514000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:21.965 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:21.975+0000 7f51337fe700 1 -- 172.21.15.139:0/3423953575 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f513c0a42b0 con 0x7f514c0f6680 2021-11-20T06:07:22.147 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:22.161+0000 7f51513d7700 1 -- 172.21.15.139:0/3423953575 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f514c027a00 con 0x7f514c0f6680 2021-11-20T06:07:22.147 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:22.161+0000 7f51337fe700 1 -- 172.21.15.139:0/3423953575 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f513c04a820 con 0x7f514c0f6680 2021-11-20T06:07:22.148 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:22.148 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:17.312170+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24327":{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":52,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:22.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:22.163+0000 7f51513d7700 1 -- 172.21.15.139:0/3423953575 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f513405c2f0 msgr2=0x7f513405e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:22.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:22.163+0000 7f51513d7700 1 --2- 172.21.15.139:0/3423953575 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f513405c2f0 0x7f513405e7b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f5140004a70 tx=0x7f514000b040).stop 2021-11-20T06:07:22.150 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:22.163+0000 7f51513d7700 1 -- 172.21.15.139:0/3423953575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f514c0f6680 msgr2=0x7f514c06b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:22.150 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:22.163+0000 7f51513d7700 1 --2- 172.21.15.139:0/3423953575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f514c0f6680 0x7f514c06b7c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f513c004e30 tx=0x7f513c009b30).stop 2021-11-20T06:07:22.150 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:22.163+0000 7f51513d7700 1 -- 172.21.15.139:0/3423953575 shutdown_connections 2021-11-20T06:07:22.150 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:22.163+0000 7f51513d7700 1 --2- 172.21.15.139:0/3423953575 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f513405c2f0 0x7f513405e7b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:22.151 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:22.163+0000 7f51513d7700 1 --2- 172.21.15.139:0/3423953575 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f514c0f6680 0x7f514c06b7c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:22.151 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:22.163+0000 7f51513d7700 1 --2- 172.21.15.139:0/3423953575 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f514c0f7920 0x7f514c069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:22.151 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:22.163+0000 7f51513d7700 1 -- 172.21.15.139:0/3423953575 >> 172.21.15.139:0/3423953575 conn(0x7f514c0f1de0 msgr2=0x7f514c0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:22.151 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:22.163+0000 7f51513d7700 1 -- 172.21.15.139:0/3423953575 shutdown_connections 2021-11-20T06:07:22.152 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:22.163+0000 7f51513d7700 1 -- 172.21.15.139:0/3423953575 wait complete. 2021-11-20T06:07:22.152 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 72 2021-11-20T06:07:22.161 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:17.312170+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24327': {'gid': 24327, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 52, 'addr': '172.21.15.178:6825/3308879813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 3308879813}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 3308879813}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:22.161 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24327:up:rejoin 2021-11-20T06:07:22.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:22 smithi178 conmon[36600]: audit 2021-11-20T06:07:22.163399+0000 mon.smithi139 (mon.0) 797 : audit [DBG] from='client.? 172.21.15.139:0/3423953575' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:22.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:22 smithi139 conmon[33501]: audit 2021-11-20T06:07:22.163399+0000 mon.smithi139 (mon.0) 797 : audit [DBG] from='client.? 172.21.15.139:0/3423953575' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:22.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:22 smithi139 conmon[33501]: 2021-11-20T06:07:23.162 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:23.300 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.311+0000 7f157669f700 1 -- 172.21.15.139:0/3883319098 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15700f6a60 msgr2=0x7f15700f6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:23.300 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.311+0000 7f157669f700 1 --2- 172.21.15.139:0/3883319098 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15700f6a60 0x7f15700f6e80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f156c004660 tx=0x7f156c009b30).stop 2021-11-20T06:07:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.312+0000 7f157669f700 1 -- 172.21.15.139:0/3883319098 shutdown_connections 2021-11-20T06:07:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.312+0000 7f157669f700 1 --2- 172.21.15.139:0/3883319098 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15700f7400 0x7f15700f5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:07:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.312+0000 7f157669f700 1 --2- 172.21.15.139:0/3883319098 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15700f6a60 0x7f15700f6e80 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:23.301 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.312+0000 7f157669f700 1 -- 172.21.15.139:0/3883319098 >> 172.21.15.139:0/3883319098 conn(0x7f15700f0360 msgr2=0x7f15700f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.312+0000 7f157669f700 1 -- 172.21.15.139:0/3883319098 shutdown_connections 2021-11-20T06:07:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.312+0000 7f157669f700 1 -- 172.21.15.139:0/3883319098 wait complete. 2021-11-20T06:07:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.312+0000 7f157669f700 1 Processor -- start 2021-11-20T06:07:23.302 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.312+0000 7f157669f700 1 -- start start 2021-11-20T06:07:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.312+0000 7f157669f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15700f6a60 0x7f1570109aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.312+0000 7f157669f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15700f7400 0x7f157010bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.312+0000 7f157669f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15700f5570 con 0x7f15700f7400 2021-11-20T06:07:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.312+0000 7f157669f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15700f56b0 con 0x7f15700f6a60 2021-11-20T06:07:23.303 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.312+0000 7f1574e9c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15700f7400 0x7f157010bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.312+0000 7f1574e9c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15700f7400 0x7f157010bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58748/0 (socket says 172.21.15.139:58748) 2021-11-20T06:07:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.312+0000 7f1574e9c700 1 -- 172.21.15.139:0/2449700442 learned_addr learned my addr 172.21.15.139:0/2449700442 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.313+0000 7f1574e9c700 1 -- 172.21.15.139:0/2449700442 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15700f6a60 msgr2=0x7f1570109aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:23.304 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.313+0000 7f157569d700 1 --2- 172.21.15.139:0/2449700442 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15700f6a60 0x7f1570109aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.313+0000 7f1574e9c700 1 --2- 172.21.15.139:0/2449700442 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15700f6a60 0x7f1570109aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.313+0000 7f1574e9c700 1 -- 172.21.15.139:0/2449700442 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156c005040 con 0x7f15700f7400 2021-11-20T06:07:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.313+0000 7f1574e9c700 1 --2- 172.21.15.139:0/2449700442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15700f7400 0x7f157010bfa0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f15600093f0 tx=0x7f1560006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:23.305 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.313+0000 7f1565ffb700 1 -- 172.21.15.139:0/2449700442 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1560013640 con 0x7f15700f7400 2021-11-20T06:07:23.306 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.313+0000 7f157669f700 1 -- 172.21.15.139:0/2449700442 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157010c5b0 con 0x7f15700f7400 2021-11-20T06:07:23.306 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.313+0000 7f1565ffb700 1 -- 172.21.15.139:0/2449700442 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1560009e40 con 0x7f15700f7400 2021-11-20T06:07:23.307 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.313+0000 7f1565ffb700 1 -- 172.21.15.139:0/2449700442 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f156000dbf0 con 0x7f15700f7400 2021-11-20T06:07:23.307 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.313+0000 7f157669f700 1 -- 172.21.15.139:0/2449700442 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157010cb70 con 0x7f15700f7400 2021-11-20T06:07:23.308 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.314+0000 7f157669f700 1 -- 172.21.15.139:0/2449700442 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1570045ad0 con 0x7f15700f7400 2021-11-20T06:07:23.308 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.314+0000 7f1565ffb700 1 -- 172.21.15.139:0/2449700442 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f156001a060 con 0x7f15700f7400 2021-11-20T06:07:23.311 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.315+0000 7f1565ffb700 1 --2- 172.21.15.139:0/2449700442 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f155c05c300 0x7f155c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:23.311 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.315+0000 7f1565ffb700 1 -- 172.21.15.139:0/2449700442 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f1560076370 con 0x7f15700f7400 2021-11-20T06:07:23.311 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.315+0000 7f157569d700 1 --2- 172.21.15.139:0/2449700442 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f155c05c300 0x7f155c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:23.312 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.315+0000 7f157569d700 1 --2- 172.21.15.139:0/2449700442 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f155c05c300 0x7f155c05e7c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f156c009d60 tx=0x7f156c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:23.312 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.319+0000 7f1565ffb700 1 -- 172.21.15.139:0/2449700442 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1560048b70 con 0x7f15700f7400 2021-11-20T06:07:23.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.503+0000 7f157669f700 1 -- 172.21.15.139:0/2449700442 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1570027a00 con 0x7f15700f7400 2021-11-20T06:07:23.489 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.503+0000 7f1565ffb700 1 -- 172.21.15.139:0/2449700442 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f156004c190 con 0x7f15700f7400 2021-11-20T06:07:23.491 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:23.491 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:17.312170+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24327":{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":52,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:23.493 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.505+0000 7f157669f700 1 -- 172.21.15.139:0/2449700442 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f155c05c300 msgr2=0x7f155c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:23.493 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.505+0000 7f157669f700 1 --2- 172.21.15.139:0/2449700442 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f155c05c300 0x7f155c05e7c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f156c009d60 tx=0x7f156c009b30).stop 2021-11-20T06:07:23.493 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.505+0000 7f157669f700 1 -- 172.21.15.139:0/2449700442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15700f7400 msgr2=0x7f157010bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:23.494 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.505+0000 7f157669f700 1 --2- 172.21.15.139:0/2449700442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15700f7400 0x7f157010bfa0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f15600093f0 tx=0x7f1560006ec0).stop 2021-11-20T06:07:23.494 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.505+0000 7f157669f700 1 -- 172.21.15.139:0/2449700442 shutdown_connections 2021-11-20T06:07:23.494 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.505+0000 7f157669f700 1 --2- 172.21.15.139:0/2449700442 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f155c05c300 0x7f155c05e7c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:23.494 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.505+0000 7f157669f700 1 --2- 172.21.15.139:0/2449700442 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15700f6a60 0x7f1570109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:23.495 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.505+0000 7f157669f700 1 --2- 172.21.15.139:0/2449700442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f15700f7400 0x7f157010bfa0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:23.495 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.505+0000 7f157669f700 1 -- 172.21.15.139:0/2449700442 >> 172.21.15.139:0/2449700442 conn(0x7f15700f0360 msgr2=0x7f15700fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:23.495 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.505+0000 7f157669f700 1 -- 172.21.15.139:0/2449700442 shutdown_connections 2021-11-20T06:07:23.495 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:23.505+0000 7f157669f700 1 -- 172.21.15.139:0/2449700442 wait complete. 2021-11-20T06:07:23.495 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 72 2021-11-20T06:07:23.501 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:23 smithi139 conmon[33501]: cluster 2021-11-20T06:07:22.270486+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:07:23.501 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:23 smithi139 conmon[33501]: mgr.14162) 307 : cluster [DBG] pgmap v281: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:23.503 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:17.312170+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24327': {'gid': 24327, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 52, 'addr': '172.21.15.178:6825/3308879813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 3308879813}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 3308879813}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:23.503 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24327:up:rejoin 2021-11-20T06:07:23.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:23 smithi178 conmon[36600]: cluster 2021-11-20T06:07:22.270486+0000 mgr.smithi139.dplyxn (mgr.14162) 307 : cluster [DBG] pgmap v281: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:24.504 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:24.632 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.645+0000 7fba5ed1f700 1 -- 172.21.15.139:0/3426147285 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba580f6510 msgr2=0x7fba580f6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:24.632 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.645+0000 7fba5ed1f700 1 --2- 172.21.15.139:0/3426147285 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba580f6510 0x7fba580f6970 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fba44004660 tx=0x7fba44009b30).stop 2021-11-20T06:07:24.632 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.645+0000 7fba5ed1f700 1 -- 172.21.15.139:0/3426147285 shutdown_connections 2021-11-20T06:07:24.633 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.645+0000 7fba5ed1f700 1 --2- 172.21.15.139:0/3426147285 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba580f6510 0x7fba580f6970 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:24.633 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.645+0000 7fba5ed1f700 1 --2- 172.21.15.139:0/3426147285 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba580f5b70 0x7fba580f5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:24.633 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.645+0000 7fba5ed1f700 1 -- 172.21.15.139:0/3426147285 >> 172.21.15.139:0/3426147285 conn(0x7fba580f0440 msgr2=0x7fba580f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:24.633 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.645+0000 7fba5ed1f700 1 -- 172.21.15.139:0/3426147285 shutdown_connections 2021-11-20T06:07:24.634 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.645+0000 7fba5ed1f700 1 -- 172.21.15.139:0/3426147285 wait complete. 2021-11-20T06:07:24.634 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.645+0000 7fba5ed1f700 1 Processor -- start 2021-11-20T06:07:24.634 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.645+0000 7fba5ed1f700 1 -- start start 2021-11-20T06:07:24.634 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba5ed1f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba580f5b70 0x7fba580f53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:24.634 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba5ed1f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba580f6510 0x7fba5810bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:24.635 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba5ed1f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5810c110 con 0x7fba580f5b70 2021-11-20T06:07:24.635 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba5ed1f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5810c250 con 0x7fba580f6510 2021-11-20T06:07:24.635 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba5cabb700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba580f5b70 0x7fba580f53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:24.635 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba5cabb700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba580f5b70 0x7fba580f53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58754/0 (socket says 172.21.15.139:58754) 2021-11-20T06:07:24.636 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba5cabb700 1 -- 172.21.15.139:0/2368243828 learned_addr learned my addr 172.21.15.139:0/2368243828 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:24.636 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba57fff700 1 --2- 172.21.15.139:0/2368243828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba580f6510 0x7fba5810bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:24.636 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba5cabb700 1 -- 172.21.15.139:0/2368243828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba580f6510 msgr2=0x7fba5810bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:24.636 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba5cabb700 1 --2- 172.21.15.139:0/2368243828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba580f6510 0x7fba5810bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:24.636 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba5cabb700 1 -- 172.21.15.139:0/2368243828 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba44005040 con 0x7fba580f5b70 2021-11-20T06:07:24.637 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba5cabb700 1 --2- 172.21.15.139:0/2368243828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba580f5b70 0x7fba580f53b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fba4c002700 tx=0x7fba4c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:24.637 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba557fa700 1 -- 172.21.15.139:0/2368243828 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba4c009320 con 0x7fba580f5b70 2021-11-20T06:07:24.637 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba5ed1f700 1 -- 172.21.15.139:0/2368243828 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5810c4e0 con 0x7fba580f5b70 2021-11-20T06:07:24.637 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba557fa700 1 -- 172.21.15.139:0/2368243828 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba4c00f040 con 0x7fba580f5b70 2021-11-20T06:07:24.638 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.646+0000 7fba557fa700 1 -- 172.21.15.139:0/2368243828 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba4c014900 con 0x7fba580f5b70 2021-11-20T06:07:24.639 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.647+0000 7fba5ed1f700 1 -- 172.21.15.139:0/2368243828 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5810caa0 con 0x7fba580f5b70 2021-11-20T06:07:24.639 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.647+0000 7fba5ed1f700 1 -- 172.21.15.139:0/2368243828 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba58027a00 con 0x7fba580f5b70 2021-11-20T06:07:24.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.648+0000 7fba557fa700 1 -- 172.21.15.139:0/2368243828 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fba4c009480 con 0x7fba580f5b70 2021-11-20T06:07:24.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.648+0000 7fba557fa700 1 --2- 172.21.15.139:0/2368243828 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fba4805c2a0 0x7fba4805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:24.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.648+0000 7fba557fa700 1 -- 172.21.15.139:0/2368243828 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fba4c076670 con 0x7fba580f5b70 2021-11-20T06:07:24.641 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.648+0000 7fba57fff700 1 --2- 172.21.15.139:0/2368243828 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fba4805c2a0 0x7fba4805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:24.642 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.649+0000 7fba57fff700 1 --2- 172.21.15.139:0/2368243828 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fba4805c2a0 0x7fba4805e760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fba44004e30 tx=0x7fba44009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:24.642 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.652+0000 7fba557fa700 1 -- 172.21.15.139:0/2368243828 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba4c048ed0 con 0x7fba580f5b70 2021-11-20T06:07:24.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:24 smithi178 conmon[36600]: audit 2021-11-20T06:07:23.505201+0000 mon.smithi139 (mon.0) 798 : audit [DBG] from='client.? 172.21.15.139:0/2449700442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:24.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:24 smithi139 conmon[33501]: audit 2021-11-20T06:07:23.505201+0000 mon.smithi139 (mon.0) 798 2021-11-20T06:07:24.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:24 smithi139 conmon[33501]: : audit [DBG] from='client.? 172.21.15.139:0/2449700442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:24.821 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.837+0000 7fba5ed1f700 1 -- 172.21.15.139:0/2368243828 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba580fdd30 con 0x7fba580f5b70 2021-11-20T06:07:24.822 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.838+0000 7fba557fa700 1 -- 172.21.15.139:0/2368243828 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2188 (secure 0 0 0) 0x7fba4c04c4f0 con 0x7fba580f5b70 2021-11-20T06:07:24.823 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:24.823 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:17.312170+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24327":{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":52,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:24.824 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.839+0000 7fba5ed1f700 1 -- 172.21.15.139:0/2368243828 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fba4805c2a0 msgr2=0x7fba4805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:24.825 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.839+0000 7fba5ed1f700 1 --2- 172.21.15.139:0/2368243828 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fba4805c2a0 0x7fba4805e760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fba44004e30 tx=0x7fba44009b30).stop 2021-11-20T06:07:24.825 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.839+0000 7fba5ed1f700 1 -- 172.21.15.139:0/2368243828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba580f5b70 msgr2=0x7fba580f53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:24.825 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.839+0000 7fba5ed1f700 1 --2- 172.21.15.139:0/2368243828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba580f5b70 0x7fba580f53b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fba4c002700 tx=0x7fba4c009d80).stop 2021-11-20T06:07:24.825 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.839+0000 7fba5ed1f700 1 -- 172.21.15.139:0/2368243828 shutdown_connections 2021-11-20T06:07:24.826 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.839+0000 7fba5ed1f700 1 --2- 172.21.15.139:0/2368243828 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fba4805c2a0 0x7fba4805e760 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:24.826 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.839+0000 7fba5ed1f700 1 --2- 172.21.15.139:0/2368243828 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fba580f5b70 0x7fba580f53b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:24.826 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.839+0000 7fba5ed1f700 1 --2- 172.21.15.139:0/2368243828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba580f6510 0x7fba5810bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:24.826 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.839+0000 7fba5ed1f700 1 -- 172.21.15.139:0/2368243828 >> 172.21.15.139:0/2368243828 conn(0x7fba580f0440 msgr2=0x7fba580fa410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:24.826 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.839+0000 7fba5ed1f700 1 -- 172.21.15.139:0/2368243828 shutdown_connections 2021-11-20T06:07:24.827 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:24.839+0000 7fba5ed1f700 1 -- 172.21.15.139:0/2368243828 wait complete. 2021-11-20T06:07:24.827 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 72 2021-11-20T06:07:24.836 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:17.312170+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24327': {'gid': 24327, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 52, 'addr': '172.21.15.178:6825/3308879813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 3308879813}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 3308879813}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:24.837 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24327:up:rejoin 2021-11-20T06:07:25.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:25 smithi178 conmon[36600]: cluster 2021-11-20T06:07:24.270895+0000 mgr.smithi139.dplyxn (mgr.14162) 308 : cluster [DBG] pgmap v282: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:25.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:25 smithi178 conmon[36600]: audit 2021-11-20T06:07:24 2021-11-20T06:07:25.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:25 smithi178 conmon[36600]: .838455+0000 mon.smithi139 (mon.0) 799 : audit [DBG] from='client.? 172.21.15.139:0/2368243828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:25.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:25 smithi139 conmon[33501]: cluster 2021-11-20T06:07:24.270895+0000 mgr.smithi139.dplyxn (mgr.14162) 308 : 2021-11-20T06:07:25.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:25 smithi139 conmon[33501]: cluster [DBG] pgmap v282: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:25.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:25 smithi139 conmon[33501]: audit 2021-11-20T06:07:24.838455+0000 mon.smithi139 (mon.0) 799 : audit [DBG] from='client.? 172.21.15.139:0/2368243828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:26.021 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:26.149 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.163+0000 7f86e4b84700 1 -- 172.21.15.139:0/4150437033 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86e00f5570 msgr2=0x7f86e00f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:26.150 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.163+0000 7f86e4b84700 1 --2- 172.21.15.139:0/4150437033 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86e00f5570 0x7f86e00f5990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f86d00099e0 tx=0x7f86d0004d10).stop 2021-11-20T06:07:26.150 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.163+0000 7f86e4b84700 1 -- 172.21.15.139:0/4150437033 shutdown_connections 2021-11-20T06:07:26.151 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.163+0000 7f86e4b84700 1 --2- 172.21.15.139:0/4150437033 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86e00f6810 0x7f86e00f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:26.151 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.163+0000 7f86e4b84700 1 --2- 172.21.15.139:0/4150437033 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86e00f5570 0x7f86e00f5990 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:26.151 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.163+0000 7f86e4b84700 1 -- 172.21.15.139:0/4150437033 >> 172.21.15.139:0/4150437033 conn(0x7f86e00f0cf0 msgr2=0x7f86e00f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:26.151 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.163+0000 7f86e4b84700 1 -- 172.21.15.139:0/4150437033 shutdown_connections 2021-11-20T06:07:26.152 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.163+0000 7f86e4b84700 1 -- 172.21.15.139:0/4150437033 wait complete. 2021-11-20T06:07:26.152 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86e4b84700 1 Processor -- start 2021-11-20T06:07:26.152 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86e4b84700 1 -- start start 2021-11-20T06:07:26.152 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86e4b84700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86e00f5570 0x7f86e00604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:26.152 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86e4b84700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86e00f6810 0x7f86e010bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:26.153 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86e4b84700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e010c290 con 0x7f86e00f5570 2021-11-20T06:07:26.153 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86e4b84700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e010c3d0 con 0x7f86e00f6810 2021-11-20T06:07:26.153 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86de59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86e00f5570 0x7f86e00604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:26.153 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86de59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86e00f5570 0x7f86e00604f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58764/0 (socket says 172.21.15.139:58764) 2021-11-20T06:07:26.154 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86de59c700 1 -- 172.21.15.139:0/2798915921 learned_addr learned my addr 172.21.15.139:0/2798915921 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:26.154 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86de59c700 1 -- 172.21.15.139:0/2798915921 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86e00f6810 msgr2=0x7f86e010bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:26.154 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86de59c700 1 --2- 172.21.15.139:0/2798915921 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86e00f6810 0x7f86e010bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:07:26.154 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86de59c700 1 -- 172.21.15.139:0/2798915921 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d0009640 con 0x7f86e00f5570 2021-11-20T06:07:26.154 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86de59c700 1 --2- 172.21.15.139:0/2798915921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86e00f5570 0x7f86e00604f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f86d0007830 tx=0x7f86d0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:26.155 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86d6ffd700 1 -- 172.21.15.139:0/2798915921 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86d000e030 con 0x7f86e00f5570 2021-11-20T06:07:26.155 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86e4b84700 1 -- 172.21.15.139:0/2798915921 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e010c5d0 con 0x7f86e00f5570 2021-11-20T06:07:26.155 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86d6ffd700 1 -- 172.21.15.139:0/2798915921 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86d001d8d0 con 0x7f86e00f5570 2021-11-20T06:07:26.155 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86d6ffd700 1 -- 172.21.15.139:0/2798915921 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86d0018070 con 0x7f86e00f5570 2021-11-20T06:07:26.155 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.164+0000 7f86e4b84700 1 -- 172.21.15.139:0/2798915921 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e010ca80 con 0x7f86e00f5570 2021-11-20T06:07:26.156 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.165+0000 7f86e4b84700 1 -- 172.21.15.139:0/2798915921 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e0046330 con 0x7f86e00f5570 2021-11-20T06:07:26.157 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.166+0000 7f86d6ffd700 1 -- 172.21.15.139:0/2798915921 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f86d001da40 con 0x7f86e00f5570 2021-11-20T06:07:26.157 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.166+0000 7f86d6ffd700 1 --2- 172.21.15.139:0/2798915921 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f86cc05c2a0 0x7f86cc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:26.158 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.166+0000 7f86d6ffd700 1 -- 172.21.15.139:0/2798915921 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f86d0078270 con 0x7f86e00f5570 2021-11-20T06:07:26.158 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.166+0000 7f86ddd9b700 1 --2- 172.21.15.139:0/2798915921 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f86cc05c2a0 0x7f86cc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:26.158 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.167+0000 7f86ddd9b700 1 --2- 172.21.15.139:0/2798915921 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f86cc05c2a0 0x7f86cc05e760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f86c80047d0 tx=0x7f86c8007a40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:26.158 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.170+0000 7f86d6ffd700 1 -- 172.21.15.139:0/2798915921 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86d004aad0 con 0x7f86e00f5570 2021-11-20T06:07:26.360 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.376+0000 7f86e4b84700 1 -- 172.21.15.139:0/2798915921 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86e00fdd00 con 0x7f86e00f5570 2021-11-20T06:07:26.361 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.376+0000 7f86d6ffd700 1 -- 172.21.15.139:0/2798915921 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f86d004e0f0 con 0x7f86e00f5570 2021-11-20T06:07:26.362 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:26.362 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:17.312170+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24327":{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":52,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:26.363 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.378+0000 7f86e4b84700 1 -- 172.21.15.139:0/2798915921 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f86cc05c2a0 msgr2=0x7f86cc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:26.363 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.378+0000 7f86e4b84700 1 --2- 172.21.15.139:0/2798915921 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f86cc05c2a0 0x7f86cc05e760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f86c80047d0 tx=0x7f86c8007a40).stop 2021-11-20T06:07:26.364 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.378+0000 7f86e4b84700 1 -- 172.21.15.139:0/2798915921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86e00f5570 msgr2=0x7f86e00604f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:26.364 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.378+0000 7f86e4b84700 1 --2- 172.21.15.139:0/2798915921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86e00f5570 0x7f86e00604f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f86d0007830 tx=0x7f86d0004d10).stop 2021-11-20T06:07:26.364 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.378+0000 7f86e4b84700 1 -- 172.21.15.139:0/2798915921 shutdown_connections 2021-11-20T06:07:26.364 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.378+0000 7f86e4b84700 1 --2- 172.21.15.139:0/2798915921 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f86cc05c2a0 0x7f86cc05e760 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:26.365 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.378+0000 7f86e4b84700 1 --2- 172.21.15.139:0/2798915921 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86e00f5570 0x7f86e00604f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:26.365 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.378+0000 7f86e4b84700 1 --2- 172.21.15.139:0/2798915921 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86e00f6810 0x7f86e010bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:07:26.365 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.378+0000 7f86e4b84700 1 -- 172.21.15.139:0/2798915921 >> 172.21.15.139:0/2798915921 conn(0x7f86e00f0cf0 msgr2=0x7f86e00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:26.365 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.378+0000 7f86e4b84700 1 -- 172.21.15.139:0/2798915921 shutdown_connections 2021-11-20T06:07:26.365 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:26.378+0000 7f86e4b84700 1 -- 172.21.15.139:0/2798915921 wait complete. 2021-11-20T06:07:26.366 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 72 2021-11-20T06:07:26.375 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:17.312170+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24327': {'gid': 24327, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 52, 'addr': '172.21.15.178:6825/3308879813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 3308879813}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 3308879813}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:26.375 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24327:up:rejoin 2021-11-20T06:07:27.376 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:27.504 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.517+0000 7f996740e700 1 -- 172.21.15.139:0/1680157065 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99600f6290 msgr2=0x7f99600f66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:27.505 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.517+0000 7f996740e700 1 --2- 172.21.15.139:0/1680157065 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99600f6290 0x7f99600f66f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f9950004660 tx=0x7f9950009b30).stop 2021-11-20T06:07:27.505 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.517+0000 7f996740e700 1 -- 172.21.15.139:0/1680157065 shutdown_connections 2021-11-20T06:07:27.505 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.517+0000 7f996740e700 1 --2- 172.21.15.139:0/1680157065 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99600f6290 0x7f99600f66f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:27.505 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.517+0000 7f996740e700 1 --2- 172.21.15.139:0/1680157065 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99600f7c80 0x7f99600f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:27.506 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.517+0000 7f996740e700 1 -- 172.21.15.139:0/1680157065 >> 172.21.15.139:0/1680157065 conn(0x7f99600f1d10 msgr2=0x7f99600f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:27.506 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.517+0000 7f996740e700 1 -- 172.21.15.139:0/1680157065 shutdown_connections 2021-11-20T06:07:27.506 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.517+0000 7f996740e700 1 -- 172.21.15.139:0/1680157065 wait complete. 2021-11-20T06:07:27.507 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.518+0000 7f996740e700 1 Processor -- start 2021-11-20T06:07:27.507 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.518+0000 7f996740e700 1 -- start start 2021-11-20T06:07:27.507 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.518+0000 7f996740e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99600f6290 0x7f9960106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:27.508 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.518+0000 7f996740e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99600f7c80 0x7f9960107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:27.508 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.518+0000 7f996740e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9960107940 con 0x7f99600f6290 2021-11-20T06:07:27.509 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.518+0000 7f996740e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9960108140 con 0x7f99600f7c80 2021-11-20T06:07:27.509 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.518+0000 7f996640c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99600f6290 0x7f9960106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:27.510 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.518+0000 7f996640c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99600f6290 0x7f9960106ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58776/0 (socket says 172.21.15.139:58776) 2021-11-20T06:07:27.510 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.518+0000 7f996640c700 1 -- 172.21.15.139:0/2466060990 learned_addr learned my addr 172.21.15.139:0/2466060990 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:27.510 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.518+0000 7f9965c0b700 1 --2- 172.21.15.139:0/2466060990 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99600f7c80 0x7f9960107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:27.510 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.518+0000 7f996640c700 1 -- 172.21.15.139:0/2466060990 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99600f7c80 msgr2=0x7f9960107390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:27.511 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.518+0000 7f996640c700 1 --2- 172.21.15.139:0/2466060990 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99600f7c80 0x7f9960107390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:27.511 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.519+0000 7f996640c700 1 -- 172.21.15.139:0/2466060990 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9950005040 con 0x7f99600f6290 2021-11-20T06:07:27.511 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.519+0000 7f9965c0b700 1 --2- 172.21.15.139:0/2466060990 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99600f7c80 0x7f9960107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:07:27.511 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.519+0000 7f996640c700 1 --2- 172.21.15.139:0/2466060990 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99600f6290 0x7f9960106ea0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f995c002700 tx=0x7f995c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:27.511 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.519+0000 7f9956ffd700 1 -- 172.21.15.139:0/2466060990 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f995c009320 con 0x7f99600f6290 2021-11-20T06:07:27.512 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.519+0000 7f9956ffd700 1 -- 172.21.15.139:0/2466060990 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f995c00f040 con 0x7f99600f6290 2021-11-20T06:07:27.512 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.519+0000 7f996740e700 1 -- 172.21.15.139:0/2466060990 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9960108420 con 0x7f99600f6290 2021-11-20T06:07:27.512 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.519+0000 7f9956ffd700 1 -- 172.21.15.139:0/2466060990 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f995c014900 con 0x7f99600f6290 2021-11-20T06:07:27.512 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.519+0000 7f996740e700 1 -- 172.21.15.139:0/2466060990 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99601089e0 con 0x7f99600f6290 2021-11-20T06:07:27.513 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.520+0000 7f996740e700 1 -- 172.21.15.139:0/2466060990 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99600f59c0 con 0x7f99600f6290 2021-11-20T06:07:27.514 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.520+0000 7f9956ffd700 1 -- 172.21.15.139:0/2466060990 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f995c009480 con 0x7f99600f6290 2021-11-20T06:07:27.514 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.521+0000 7f9956ffd700 1 --2- 172.21.15.139:0/2466060990 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f994c05c300 0x7f994c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:27.514 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.521+0000 7f9956ffd700 1 -- 172.21.15.139:0/2466060990 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f995c076940 con 0x7f99600f6290 2021-11-20T06:07:27.515 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.521+0000 7f9965c0b700 1 --2- 172.21.15.139:0/2466060990 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f994c05c300 0x7f994c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:27.515 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.521+0000 7f9965c0b700 1 --2- 172.21.15.139:0/2466060990 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f994c05c300 0x7f994c05e7c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f9950005010 tx=0x7f9950009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:27.515 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.524+0000 7f9956ffd700 1 -- 172.21.15.139:0/2466060990 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f995c049220 con 0x7f99600f6290 2021-11-20T06:07:27.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:27 smithi178 conmon[36600]: cluster 2021-11-20T06:07:26.271292+0000 mgr.smithi139.dplyxn (mgr.14162) 309 : cluster [DBG] pgmap v283: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:07:27.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:27 smithi178 conmon[36600]: audit 2021-11-20T06:07:26.377196+0000 mon.smithi139 (mon.0) 800 : audit [DBG] from='client.? 172.21.15.139:0/2798915921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:27.694 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.709+0000 7f996740e700 1 -- 172.21.15.139:0/2466060990 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9960045ad0 con 0x7f99600f6290 2021-11-20T06:07:27.694 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.709+0000 7f9956ffd700 1 -- 172.21.15.139:0/2466060990 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f995c04c840 con 0x7f99600f6290 2021-11-20T06:07:27.696 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:27.696 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:17.312170+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24327":{"gid":24327,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":52,"addr":"172.21.15.178:6825/3308879813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":3308879813},{"type":"v1","addr":"172.21.15.178:6825","nonce":3308879813}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:27.699 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.711+0000 7f996740e700 1 -- 172.21.15.139:0/2466060990 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f994c05c300 msgr2=0x7f994c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:27.699 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.711+0000 7f996740e700 1 --2- 172.21.15.139:0/2466060990 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f994c05c300 0x7f994c05e7c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f9950005010 tx=0x7f9950009b30).stop 2021-11-20T06:07:27.700 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.711+0000 7f996740e700 1 -- 172.21.15.139:0/2466060990 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99600f6290 msgr2=0x7f9960106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:27.700 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.711+0000 7f996740e700 1 --2- 172.21.15.139:0/2466060990 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99600f6290 0x7f9960106ea0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f995c002700 tx=0x7f995c009d80).stop 2021-11-20T06:07:27.700 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.711+0000 7f996740e700 1 -- 172.21.15.139:0/2466060990 shutdown_connections 2021-11-20T06:07:27.701 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.711+0000 7f996740e700 1 --2- 172.21.15.139:0/2466060990 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f994c05c300 0x7f994c05e7c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:27.701 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.711+0000 7f996740e700 1 --2- 172.21.15.139:0/2466060990 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99600f6290 0x7f9960106ea0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.711+0000 7f996740e700 1 --2- 172.21.15.139:0/2466060990 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99600f7c80 0x7f9960107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.711+0000 7f996740e700 1 -- 172.21.15.139:0/2466060990 >> 172.21.15.139:0/2466060990 conn(0x7f99600f1d10 msgr2=0x7f99600fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.711+0000 7f996740e700 1 -- 172.21.15.139:0/2466060990 shutdown_connections 2021-11-20T06:07:27.702 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:27.711+0000 7f996740e700 1 -- 172.21.15.139:0/2466060990 wait complete. 2021-11-20T06:07:27.703 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 72 2021-11-20T06:07:27.706 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:27 smithi139 conmon[33501]: cluster 2021-11-20T06:07:26.271292+0000 mgr.smithi139.dplyxn (mgr.14162) 309 : cluster [DBG] pgmap v283: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:07:27.707 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:27 smithi139 conmon[33501]: audit 2021-11-20T06:07:26.377196+0000 mon.smithi139 (mon.0) 800 : audit [DBG] from='client.? 172.21.15.139:0/2798915921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:27.708 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:17.312170+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24327': {'gid': 24327, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 52, 'addr': '172.21.15.178:6825/3308879813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 3308879813}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 3308879813}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:27.708 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24327:up:rejoin 2021-11-20T06:07:28.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:28 smithi178 conmon[36600]: audit 2021-11-20T06:07:27. 2021-11-20T06:07:28.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:28 smithi178 conmon[36600]: 710258+0000 mon.smithi139 (mon.0) 801 : audit [DBG] from='client.? 172.21.15.139:0/2466060990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:28.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:28 smithi178 conmon[36600]: cluster 2021-11-20T06:07:28.187598+0000 mon.smithi139 (mon 2021-11-20T06:07:28.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:28 smithi178 conmon[36600]: .0) 802 : cluster [INF] daemon mds.cephfs.smithi178.blvkbx restarted 2021-11-20T06:07:28.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:28 smithi178 conmon[36600]: cluster 2021-11- 2021-11-20T06:07:28.442 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:28 smithi178 conmon[36600]: 20T06:07:28.191376+0000 2021-11-20T06:07:28.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:28 smithi178 conmon[36600]: mon.smithi139 (mon.0) 803 : cluster 2021-11-20T06:07:28.443 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:28 smithi178 conmon[36600]: [DBG] osdmap e55: 6 total, 6 up, 6 in 2021-11-20T06:07:28.710 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:28.737 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:28 smithi139 conmon[33501]: debug 2021-11-20T06:07:28.345+0000 7f2a21654700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:07:28.738 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:28 smithi139 conmon[33501]: audit 2021-11-20T06:07:27.710258 2021-11-20T06:07:28.738 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:28 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 801 : audit [DBG] from='client.? 172.21.15.139:0/2466060990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:28.738 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:28 smithi139 conmon[33501]: cluster 2021-11-20T06:07:28.187598+0000 mon.smithi139 (mon 2021-11-20T06:07:28.738 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:28 smithi139 conmon[33501]: .0) 802 : cluster [INF] daemon mds.cephfs.smithi178.blvkbx restarted 2021-11-20T06:07:28.739 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:28 smithi139 conmon[33501]: cluster 2021-11-20T 2021-11-20T06:07:28.739 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:28 smithi139 conmon[33501]: 06:07:28.191376+0000 mon.smithi139 (mon.0) 803 : cluster [DBG] osdmap e55: 6 total, 6 up, 6 in 2021-11-20T06:07:28.841 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.855+0000 7fc298f37700 1 -- 172.21.15.139:0/3694703810 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2940f6850 msgr2=0x7fc2940f6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:28.842 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.855+0000 7fc298f37700 1 --2- 172.21.15.139:0/3694703810 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2940f6850 0x7fc2940f6cb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fc27c004660 tx=0x7fc27c009b30).stop 2021-11-20T06:07:28.842 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.855+0000 7fc298f37700 1 -- 172.21.15.139:0/3694703810 shutdown_connections 2021-11-20T06:07:28.842 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.855+0000 7fc298f37700 1 --2- 172.21.15.139:0/3694703810 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2940f6850 0x7fc2940f6cb0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:28.843 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.855+0000 7fc298f37700 1 --2- 172.21.15.139:0/3694703810 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2940f55b0 0x7fc2940f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:28.843 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.855+0000 7fc298f37700 1 -- 172.21.15.139:0/3694703810 >> 172.21.15.139:0/3694703810 conn(0x7fc2940f0d10 msgr2=0x7fc2940f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:28.843 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.855+0000 7fc298f37700 1 -- 172.21.15.139:0/3694703810 shutdown_connections 2021-11-20T06:07:28.843 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.855+0000 7fc298f37700 1 -- 172.21.15.139:0/3694703810 wait complete. 2021-11-20T06:07:28.843 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.855+0000 7fc298f37700 1 Processor -- start 2021-11-20T06:07:28.844 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc298f37700 1 -- start start 2021-11-20T06:07:28.844 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc298f37700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2940f55b0 0x7fc294106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:28.844 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc298f37700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2940f6850 0x7fc294107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:28.844 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc298f37700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc294107940 con 0x7fc2940f55b0 2021-11-20T06:07:28.844 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc298f37700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc294107a80 con 0x7fc2940f6850 2021-11-20T06:07:28.845 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc29259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2940f55b0 0x7fc294106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:28.845 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc29259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2940f55b0 0x7fc294106ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58786/0 (socket says 172.21.15.139:58786) 2021-11-20T06:07:28.845 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc29259c700 1 -- 172.21.15.139:0/936704451 learned_addr learned my addr 172.21.15.139:0/936704451 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:28.845 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc28bfff700 1 --2- 172.21.15.139:0/936704451 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2940f6850 0x7fc294107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:28.846 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc29259c700 1 -- 172.21.15.139:0/936704451 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2940f6850 msgr2=0x7fc294107390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:28.846 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc29259c700 1 --2- 172.21.15.139:0/936704451 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2940f6850 0x7fc294107390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:28.846 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc29259c700 1 -- 172.21.15.139:0/936704451 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc27c005040 con 0x7fc2940f55b0 2021-11-20T06:07:28.847 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc29259c700 1 --2- 172.21.15.139:0/936704451 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2940f55b0 0x7fc294106ea0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fc284002700 tx=0x7fc284009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:28.847 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc28affd700 1 -- 172.21.15.139:0/936704451 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2840079c0 con 0x7fc2940f55b0 2021-11-20T06:07:28.847 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc28affd700 1 -- 172.21.15.139:0/936704451 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc28400f040 con 0x7fc2940f55b0 2021-11-20T06:07:28.847 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc298f37700 1 -- 172.21.15.139:0/936704451 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29410c520 con 0x7fc2940f55b0 2021-11-20T06:07:28.848 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc28affd700 1 -- 172.21.15.139:0/936704451 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc284014860 con 0x7fc2940f55b0 2021-11-20T06:07:28.848 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.856+0000 7fc298f37700 1 -- 172.21.15.139:0/936704451 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29410cae0 con 0x7fc2940f55b0 2021-11-20T06:07:28.848 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.857+0000 7fc298f37700 1 -- 172.21.15.139:0/936704451 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2940fdd40 con 0x7fc2940f55b0 2021-11-20T06:07:28.849 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.858+0000 7fc28affd700 1 -- 172.21.15.139:0/936704451 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc28401b060 con 0x7fc2940f55b0 2021-11-20T06:07:28.850 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.858+0000 7fc28affd700 1 --2- 172.21.15.139:0/936704451 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc28005c2a0 0x7fc28005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:28.850 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.858+0000 7fc28affd700 1 -- 172.21.15.139:0/936704451 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fc284076230 con 0x7fc2940f55b0 2021-11-20T06:07:28.850 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.858+0000 7fc28bfff700 1 --2- 172.21.15.139:0/936704451 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc28005c2a0 0x7fc28005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:28.850 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.859+0000 7fc28bfff700 1 --2- 172.21.15.139:0/936704451 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc28005c2a0 0x7fc28005e760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fc27c009f50 tx=0x7fc27c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:28.851 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:28.862+0000 7fc28affd700 1 -- 172.21.15.139:0/936704451 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc284048a40 con 0x7fc2940f55b0 2021-11-20T06:07:29.029 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:29.045+0000 7fc298f37700 1 -- 172.21.15.139:0/936704451 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc29410d1e0 con 0x7fc2940f55b0 2021-11-20T06:07:29.030 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:29.045+0000 7fc28affd700 1 -- 172.21.15.139:0/936704451 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+2188 (secure 0 0 0) 0x7fc28404c060 con 0x7fc2940f55b0 2021-11-20T06:07:29.031 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:29.031 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":74,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:28.349661+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24489":{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":74,"state":"up:replay","state_seq":2,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:29.032 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:29.047+0000 7fc298f37700 1 -- 172.21.15.139:0/936704451 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc28005c2a0 msgr2=0x7fc28005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:29.033 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:29.047+0000 7fc298f37700 1 --2- 172.21.15.139:0/936704451 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc28005c2a0 0x7fc28005e760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fc27c009f50 tx=0x7fc27c009b30).stop 2021-11-20T06:07:29.033 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:29.047+0000 7fc298f37700 1 -- 172.21.15.139:0/936704451 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2940f55b0 msgr2=0x7fc294106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:29.033 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:29.047+0000 7fc298f37700 1 --2- 172.21.15.139:0/936704451 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2940f55b0 0x7fc294106ea0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fc284002700 tx=0x7fc284009d80).stop 2021-11-20T06:07:29.033 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:29.047+0000 7fc298f37700 1 -- 172.21.15.139:0/936704451 shutdown_connections 2021-11-20T06:07:29.033 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:29.047+0000 7fc298f37700 1 --2- 172.21.15.139:0/936704451 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc28005c2a0 0x7fc28005e760 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:29.034 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:29.047+0000 7fc298f37700 1 --2- 172.21.15.139:0/936704451 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc2940f55b0 0x7fc294106ea0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:29.034 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:29.047+0000 7fc298f37700 1 --2- 172.21.15.139:0/936704451 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2940f6850 0x7fc294107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:29.034 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:29.047+0000 7fc298f37700 1 -- 172.21.15.139:0/936704451 >> 172.21.15.139:0/936704451 conn(0x7fc2940f0d10 msgr2=0x7fc29405f490 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:29.035 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:29.047+0000 7fc298f37700 1 -- 172.21.15.139:0/936704451 shutdown_connections 2021-11-20T06:07:29.036 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:29.047+0000 7fc298f37700 1 -- 172.21.15.139:0/936704451 wait complete. 2021-11-20T06:07:29.036 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 74 2021-11-20T06:07:29.043 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 74, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:28.349661+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24489': {'gid': 24489, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 74, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.178:6827/2887891857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2887891857}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2887891857}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:29.044 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24489:up:replay 2021-11-20T06:07:29.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: cluster 2021-11-20T06:07:28.271769+0000 mgr.smithi139.dplyxn (mgr.14162) 310 : cluster [DBG] pgmap v285: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T06:07:29.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: cluster 2021-11-20T06:07:28.346249+0000 mon.smithi139 (mon.0) 804 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:07:29.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: cluster 2021-11-20T06:07:28.349544+0000 mon.smithi139 (mon.0) 805 : cluster [DBG] mds.? [v2:172.21.15.178:6824/1490972326,v1:172.21.15.178:6825/1490972326] up:boot 2021-11-20T06:07:29.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: cluster 2021-11-20T06:07:28.349613+0000 mon.smithi139 (mon.0) 806 : cluster [INF] Standby daemon mds.cephfs.smithi178.pskihy assigned to filesystem cephfs as rank 0 2021-11-20T06:07:29.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: cluster 2021-11-20T06:07:28 2021-11-20T06:07:29.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: .349728+0000 mon.smithi139 (mon.0) 807 : cluster [INF] 2021-11-20T06:07:29.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:07:29.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: cluster 2021-11-20T06:07:28 2021-11-20T06:07:29.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: .350029+0000 mon.smithi139 (mon.0) 808 : cluster 2021-11-20T06:07:29.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T06:07:29.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: audit 2021-11-20T06:07:28 2021-11-20T06:07:29.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: .350192+0000 mon.smithi139 (mon.0) 809 : audit 2021-11-20T06:07:29.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.blvkbx"}]: dispatch 2021-11-20T06:07:29.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: cluster 2021-11-20T06:07:28. 2021-11-20T06:07:29.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: 355678+0000 mon.smithi139 (mon.0) 810 : cluster [DBG] 2021-11-20T06:07:29.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:replay} 1 up:standby 2021-11-20T06:07:29.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: audit 2021-11-20T06:07: 2021-11-20T06:07:29.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: 29.046119+0000 mon.smithi139 (mon.0) 811 : audit 2021-11-20T06:07:29.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:29 smithi178 conmon[36600]: [DBG] from='client.? 172.21.15.139:0/936704451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:29.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: cluster 2021-11-20T06:07:28.271769+0000 mgr.smithi139.dplyxn (mgr.14162) 310 : cluster [DBG] pgmap v285: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T06:07:29.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: cluster 2021-11-20T06:07:28.346249 2021-11-20T06:07:29.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 804 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:07:29.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: cluster 2021-11-20T06:07:28.349544+0000 mon.smithi139 (mon.0) 805 : cluster [DBG] mds.? [v2:172.21.15.178:6824/1490972326,v1:172.21.15.178:6825/1490972326] up:boot 2021-11-20T06:07:29.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: cluster 2021-11-20T06: 2021-11-20T06:07:29.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: 07:28.349613+0000 mon.smithi139 (mon.0) 806 : cluster [INF] Standby daemon mds.cephfs.smithi178.pskihy assigned to filesystem cephfs as rank 0 2021-11-20T06:07:29.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: cluster 2021-11-20T06:07:28.349728+0000 mon.smithi139 (mon.0) 807 2021-11-20T06:07:29.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:07:29.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: cluster 2021-11-20T06:07:29.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: 2021-11-20 2021-11-20T06:07:29.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: T06:07:28.350029+0000 mon.smithi139 2021-11-20T06:07:29.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: (mon.0) 808 : cluster [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T06:07:29.779 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: 2021-11-20T06:07:29.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: audit 2021- 2021-11-20T06:07:29.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: 11-20T06:07:28.350192+0000 mon.smithi139 (mon.0 2021-11-20T06:07:29.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: ) 809 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.blvkbx"}]: dispatch 2021-11-20T06:07:29.780 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: cluster 2021-11-20T06:07:28.355678+0000 mon.smithi139 (mon 2021-11-20T06:07:29.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: .0) 810 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:replay} 1 up:standby 2021-11-20T06:07:29.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: audit 2021-11-20T06:07:29.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: 2021-11-20T06:07:29.046119+0000 mon.smithi139 (mon 2021-11-20T06:07:29.781 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:29 smithi139 conmon[33501]: .0) 811 : audit [DBG] from='client.? 172.21.15.139:0/936704451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:30.045 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:30.173 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.187+0000 7fd4b59f1700 1 -- 172.21.15.139:0/3205757117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4b00f7920 msgr2=0x7fd4b00f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:30.174 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.187+0000 7fd4b59f1700 1 --2- 172.21.15.139:0/3205757117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4b00f7920 0x7fd4b00f7d80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fd4a4004660 tx=0x7fd4a4009b30).stop 2021-11-20T06:07:30.174 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.187+0000 7fd4b59f1700 1 -- 172.21.15.139:0/3205757117 shutdown_connections 2021-11-20T06:07:30.174 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.187+0000 7fd4b59f1700 1 --2- 172.21.15.139:0/3205757117 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4b00f7920 0x7fd4b00f7d80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:30.174 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.187+0000 7fd4b59f1700 1 --2- 172.21.15.139:0/3205757117 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4b00f6680 0x7fd4b00f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:30.175 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.187+0000 7fd4b59f1700 1 -- 172.21.15.139:0/3205757117 >> 172.21.15.139:0/3205757117 conn(0x7fd4b00f1de0 msgr2=0x7fd4b00f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:30.175 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.187+0000 7fd4b59f1700 1 -- 172.21.15.139:0/3205757117 shutdown_connections 2021-11-20T06:07:30.175 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.187+0000 7fd4b59f1700 1 -- 172.21.15.139:0/3205757117 wait complete. 2021-11-20T06:07:30.175 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4b59f1700 1 Processor -- start 2021-11-20T06:07:30.175 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4b59f1700 1 -- start start 2021-11-20T06:07:30.176 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4b59f1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4b00f6680 0x7fd4b0106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:30.176 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4b59f1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4b00f7920 0x7fd4b01073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:30.176 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4b59f1700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b0107950 con 0x7fd4b00f6680 2021-11-20T06:07:30.176 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4b59f1700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b0107a90 con 0x7fd4b00f7920 2021-11-20T06:07:30.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4aeffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4b00f6680 0x7fd4b0106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:30.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4aeffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4b00f6680 0x7fd4b0106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58798/0 (socket says 172.21.15.139:58798) 2021-11-20T06:07:30.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4aeffd700 1 -- 172.21.15.139:0/3728405059 learned_addr learned my addr 172.21.15.139:0/3728405059 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:30.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4aeffd700 1 -- 172.21.15.139:0/3728405059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4b00f7920 msgr2=0x7fd4b01073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:30.177 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4aeffd700 1 --2- 172.21.15.139:0/3728405059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4b00f7920 0x7fd4b01073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:07:30.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4aeffd700 1 -- 172.21.15.139:0/3728405059 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a4005040 con 0x7fd4b00f6680 2021-11-20T06:07:30.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4aeffd700 1 --2- 172.21.15.139:0/3728405059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4b00f6680 0x7fd4b0106eb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fd4a0002700 tx=0x7fd4a0009d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:30.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4977fe700 1 -- 172.21.15.139:0/3728405059 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4a0009320 con 0x7fd4b00f6680 2021-11-20T06:07:30.178 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4977fe700 1 -- 172.21.15.139:0/3728405059 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4a000f040 con 0x7fd4b00f6680 2021-11-20T06:07:30.179 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4b59f1700 1 -- 172.21.15.139:0/3728405059 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b010c530 con 0x7fd4b00f6680 2021-11-20T06:07:30.179 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.188+0000 7fd4977fe700 1 -- 172.21.15.139:0/3728405059 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4a0014730 con 0x7fd4b00f6680 2021-11-20T06:07:30.179 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.189+0000 7fd4b59f1700 1 -- 172.21.15.139:0/3728405059 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b010caf0 con 0x7fd4b00f6680 2021-11-20T06:07:30.180 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.189+0000 7fd4b59f1700 1 -- 172.21.15.139:0/3728405059 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b0027a00 con 0x7fd4b00f6680 2021-11-20T06:07:30.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.190+0000 7fd4977fe700 1 -- 172.21.15.139:0/3728405059 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd4a0009480 con 0x7fd4b00f6680 2021-11-20T06:07:30.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.190+0000 7fd4977fe700 1 --2- 172.21.15.139:0/3728405059 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd49805c2a0 0x7fd49805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:30.181 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.190+0000 7fd4977fe700 1 -- 172.21.15.139:0/3728405059 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fd4a0076a00 con 0x7fd4b00f6680 2021-11-20T06:07:30.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.190+0000 7fd4ae7fc700 1 --2- 172.21.15.139:0/3728405059 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd49805c2a0 0x7fd49805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:30.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.191+0000 7fd4ae7fc700 1 --2- 172.21.15.139:0/3728405059 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd49805c2a0 0x7fd49805e760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fd4a4009f50 tx=0x7fd4a4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:30.182 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.194+0000 7fd4977fe700 1 -- 172.21.15.139:0/3728405059 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4a00491b0 con 0x7fd4b00f6680 2021-11-20T06:07:30.372 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.388+0000 7fd4b59f1700 1 -- 172.21.15.139:0/3728405059 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4b00fee30 con 0x7fd4b00f6680 2021-11-20T06:07:30.373 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.388+0000 7fd4977fe700 1 -- 172.21.15.139:0/3728405059 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+2188 (secure 0 0 0) 0x7fd4a004c7d0 con 0x7fd4b00f6680 2021-11-20T06:07:30.373 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:30.374 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":76,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:30.359623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24489":{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:30.375 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.390+0000 7fd4b59f1700 1 -- 172.21.15.139:0/3728405059 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd49805c2a0 msgr2=0x7fd49805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:30.376 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.390+0000 7fd4b59f1700 1 --2- 172.21.15.139:0/3728405059 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd49805c2a0 0x7fd49805e760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fd4a4009f50 tx=0x7fd4a4009b30).stop 2021-11-20T06:07:30.376 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.390+0000 7fd4b59f1700 1 -- 172.21.15.139:0/3728405059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4b00f6680 msgr2=0x7fd4b0106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:30.377 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.390+0000 7fd4b59f1700 1 --2- 172.21.15.139:0/3728405059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4b00f6680 0x7fd4b0106eb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fd4a0002700 tx=0x7fd4a0009d60).stop 2021-11-20T06:07:30.377 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.390+0000 7fd4b59f1700 1 -- 172.21.15.139:0/3728405059 shutdown_connections 2021-11-20T06:07:30.377 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.390+0000 7fd4b59f1700 1 --2- 172.21.15.139:0/3728405059 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd49805c2a0 0x7fd49805e760 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:30.377 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.390+0000 7fd4b59f1700 1 --2- 172.21.15.139:0/3728405059 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd4b00f6680 0x7fd4b0106eb0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:30.377 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.390+0000 7fd4b59f1700 1 --2- 172.21.15.139:0/3728405059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4b00f7920 0x7fd4b01073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:07:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.390+0000 7fd4b59f1700 1 -- 172.21.15.139:0/3728405059 >> 172.21.15.139:0/3728405059 conn(0x7fd4b00f1de0 msgr2=0x7fd4b00fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.390+0000 7fd4b59f1700 1 -- 172.21.15.139:0/3728405059 shutdown_connections 2021-11-20T06:07:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:30.390+0000 7fd4b59f1700 1 -- 172.21.15.139:0/3728405059 wait complete. 2021-11-20T06:07:30.378 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 76 2021-11-20T06:07:30.387 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:30.359623+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24489': {'gid': 24489, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.178:6827/2887891857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2887891857}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2887891857}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:30.387 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24489:up:rejoin 2021-11-20T06:07:30.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:30 smithi178 conmon[36600]: cluster 2021-11-20T06:07:29.360616+0000 mon.smithi139 (mon.0) 812 : cluster [DBG] mds.? [v2:172.21.15.178:6826/2887891857,v1:172.21.15.178:6827/2887891857] up:reconnect 2021-11-20T06:07:30.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:30 smithi178 conmon[36600]: cluster 2021-11-20T06:07:29.360697+0000 2021-11-20T06:07:30.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:30 smithi178 conmon[36600]: mon.smithi139 (mon.0) 813 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:reconnect} 1 up:standby 2021-11-20T06:07:30.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:30 smithi139 conmon[33501]: cluster 2021-11-20T06:07:29.360616+0000 mon.smithi139 (mon.0) 812 : cluster [DBG] 2021-11-20T06:07:30.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:30 smithi139 conmon[33501]: mds.? [v2:172.21.15.178:6826/2887891857,v1:172.21.15.178:6827/2887891857] up:reconnect 2021-11-20T06:07:30.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:30 smithi139 conmon[33501]: cluster 2021-11-20T06:07:29.360697+0000 mon.smithi139 (mon.0) 813 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:reconnect} 1 up:standby 2021-11-20T06:07:31.389 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:31.536 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.549+0000 7fb1cacb3700 1 -- 172.21.15.139:0/1989056728 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c40f4c20 msgr2=0x7fb1c40f5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:31.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.549+0000 7fb1cacb3700 1 --2- 172.21.15.139:0/1989056728 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c40f4c20 0x7fb1c40f5040 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fb1c0004770 tx=0x7fb1c000ac40).stop 2021-11-20T06:07:31.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.549+0000 7fb1cacb3700 1 -- 172.21.15.139:0/1989056728 shutdown_connections 2021-11-20T06:07:31.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.549+0000 7fb1cacb3700 1 --2- 172.21.15.139:0/1989056728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1c40f5ec0 0x7fb1c40f6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:31.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.549+0000 7fb1cacb3700 1 --2- 172.21.15.139:0/1989056728 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c40f4c20 0x7fb1c40f5040 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:31.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.549+0000 7fb1cacb3700 1 -- 172.21.15.139:0/1989056728 >> 172.21.15.139:0/1989056728 conn(0x7fb1c40f0360 msgr2=0x7fb1c40f27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:31.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.549+0000 7fb1cacb3700 1 -- 172.21.15.139:0/1989056728 shutdown_connections 2021-11-20T06:07:31.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.549+0000 7fb1cacb3700 1 -- 172.21.15.139:0/1989056728 wait complete. 2021-11-20T06:07:31.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.550+0000 7fb1cacb3700 1 Processor -- start 2021-11-20T06:07:31.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.550+0000 7fb1cacb3700 1 -- start start 2021-11-20T06:07:31.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.550+0000 7fb1cacb3700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c40f4c20 0x7fb1c40f8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:31.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.550+0000 7fb1cacb3700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1c40f5ec0 0x7fb1c40f7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:31.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.550+0000 7fb1cacb3700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c40f7760 con 0x7fb1c40f4c20 2021-11-20T06:07:31.540 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.550+0000 7fb1cacb3700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c40f78a0 con 0x7fb1c40f5ec0 2021-11-20T06:07:31.540 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.550+0000 7fb1c9cb1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c40f4c20 0x7fb1c40f8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:31.540 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.550+0000 7fb1c9cb1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c40f4c20 0x7fb1c40f8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58812/0 (socket says 172.21.15.139:58812) 2021-11-20T06:07:31.540 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.550+0000 7fb1c9cb1700 1 -- 172.21.15.139:0/3200950516 learned_addr learned my addr 172.21.15.139:0/3200950516 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:31.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.550+0000 7fb1c94b0700 1 --2- 172.21.15.139:0/3200950516 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1c40f5ec0 0x7fb1c40f7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:31.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.550+0000 7fb1c9cb1700 1 -- 172.21.15.139:0/3200950516 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1c40f5ec0 msgr2=0x7fb1c40f7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:31.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.550+0000 7fb1c9cb1700 1 --2- 172.21.15.139:0/3200950516 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1c40f5ec0 0x7fb1c40f7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:31.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.550+0000 7fb1c9cb1700 1 -- 172.21.15.139:0/3200950516 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c0005040 con 0x7fb1c40f4c20 2021-11-20T06:07:31.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.551+0000 7fb1c9cb1700 1 --2- 172.21.15.139:0/3200950516 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c40f4c20 0x7fb1c40f8d80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fb1c0014490 tx=0x7fb1c000ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:31.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.551+0000 7fb1ba7fc700 1 -- 172.21.15.139:0/3200950516 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1c000f030 con 0x7fb1c40f4c20 2021-11-20T06:07:31.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.551+0000 7fb1ba7fc700 1 -- 172.21.15.139:0/3200950516 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1c000a590 con 0x7fb1c40f4c20 2021-11-20T06:07:31.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.551+0000 7fb1cacb3700 1 -- 172.21.15.139:0/3200950516 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c40f7b20 con 0x7fb1c40f4c20 2021-11-20T06:07:31.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.551+0000 7fb1ba7fc700 1 -- 172.21.15.139:0/3200950516 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1c0019070 con 0x7fb1c40f4c20 2021-11-20T06:07:31.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.551+0000 7fb1cacb3700 1 -- 172.21.15.139:0/3200950516 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c40f8080 con 0x7fb1c40f4c20 2021-11-20T06:07:31.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.552+0000 7fb1cacb3700 1 -- 172.21.15.139:0/3200950516 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c4045ad0 con 0x7fb1c40f4c20 2021-11-20T06:07:31.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.552+0000 7fb1ba7fc700 1 -- 172.21.15.139:0/3200950516 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb1c00083a0 con 0x7fb1c40f4c20 2021-11-20T06:07:31.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.552+0000 7fb1ba7fc700 1 --2- 172.21.15.139:0/3200950516 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb1b005c300 0x7fb1b005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:31.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.552+0000 7fb1ba7fc700 1 -- 172.21.15.139:0/3200950516 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb1c0077ce0 con 0x7fb1c40f4c20 2021-11-20T06:07:31.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.552+0000 7fb1c94b0700 1 --2- 172.21.15.139:0/3200950516 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb1b005c300 0x7fb1b005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:31.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.553+0000 7fb1c94b0700 1 --2- 172.21.15.139:0/3200950516 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb1b005c300 0x7fb1b005e7c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fb1b4007fd0 tx=0x7fb1b400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:31.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.556+0000 7fb1ba7fc700 1 -- 172.21.15.139:0/3200950516 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1c004a570 con 0x7fb1c40f4c20 2021-11-20T06:07:31.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:31 smithi178 conmon[36600]: cluster 2021-11-20T06:07:30.272198+0000 mgr.smithi139.dplyxn (mgr.14162) 311 : cluster [DBG] pgmap v286: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T06:07:31.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:31 smithi178 conmon[36600]: cluster 2021-11-20T06:07:30.364947+0000 mon.smithi139 (mon.0) 814 : cluster [DBG] mds.? [v2:172.21.15.178:6826/2887891857,v1:172.21.15.178:6827/2887891857] up:rejoin 2021-11-20T06:07:31.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:31 smithi178 conmon[36600]: cluster 2021-11-20T06:07:30.365006+0000 mon.smithi139 (mon.0) 815 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:rejoin} 1 up:standby 2021-11-20T06:07:31.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:31 smithi178 conmon[36600]: audit 2021-11-20T06:07:30.389152+0000 mon.smithi139 (mon.0) 816 : audit [DBG] from='client.? 172.21.15.139:0/3728405059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:31.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:31 smithi178 conmon[36600]: audit 2021-11-20T06:07:31.257950+0000 mon.smithi139 (mon.0) 817 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:07:31.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:31 smithi178 conmon[36600]: audit 2021-11-20T06:07:31.259099+0000 mon.smithi139 (mon.0) 818 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:07:31.726 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.741+0000 7fb1cacb3700 1 -- 172.21.15.139:0/3200950516 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1c40fcb40 con 0x7fb1c40f4c20 2021-11-20T06:07:31.727 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.742+0000 7fb1ba7fc700 1 -- 172.21.15.139:0/3200950516 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+2188 (secure 0 0 0) 0x7fb1c004db90 con 0x7fb1c40f4c20 2021-11-20T06:07:31.728 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:31.728 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":76,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:30.359623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24489":{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:31.729 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.743+0000 7fb1cacb3700 1 -- 172.21.15.139:0/3200950516 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb1b005c300 msgr2=0x7fb1b005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:31.730 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.743+0000 7fb1cacb3700 1 --2- 172.21.15.139:0/3200950516 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb1b005c300 0x7fb1b005e7c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fb1b4007fd0 tx=0x7fb1b400b040).stop 2021-11-20T06:07:31.730 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.743+0000 7fb1cacb3700 1 -- 172.21.15.139:0/3200950516 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c40f4c20 msgr2=0x7fb1c40f8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:31.732 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.743+0000 7fb1cacb3700 1 --2- 172.21.15.139:0/3200950516 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c40f4c20 0x7fb1c40f8d80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fb1c0014490 tx=0x7fb1c000ac40).stop 2021-11-20T06:07:31.732 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.743+0000 7fb1cacb3700 1 -- 172.21.15.139:0/3200950516 shutdown_connections 2021-11-20T06:07:31.732 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.743+0000 7fb1cacb3700 1 --2- 172.21.15.139:0/3200950516 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fb1b005c300 0x7fb1b005e7c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:31.732 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.743+0000 7fb1cacb3700 1 --2- 172.21.15.139:0/3200950516 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb1c40f4c20 0x7fb1c40f8d80 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:31.733 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.743+0000 7fb1cacb3700 1 --2- 172.21.15.139:0/3200950516 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1c40f5ec0 0x7fb1c40f7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:31.733 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.743+0000 7fb1cacb3700 1 -- 172.21.15.139:0/3200950516 >> 172.21.15.139:0/3200950516 conn(0x7fb1c40f0360 msgr2=0x7fb1c40f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:31.733 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.743+0000 7fb1cacb3700 1 -- 172.21.15.139:0/3200950516 shutdown_connections 2021-11-20T06:07:31.733 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:31.743+0000 7fb1cacb3700 1 -- 172.21.15.139:0/3200950516 wait complete. 2021-11-20T06:07:31.734 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 76 2021-11-20T06:07:31.738 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:31 smithi139 conmon[33501]: cluster 2021-11-20T06:07:30.272198+0000 mgr.smithi139.dplyxn (mgr.14162) 311 : cluster [DBG] pgmap v286: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T06:07:31.739 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:31 smithi139 conmon[33501]: cluster 2021-11-20T06:07:30.364947+0000 mon.smithi139 (mon.0) 814 : cluster [DBG] mds.? [v2:172.21.15.178:6826/2887891857,v1:172.21.15.178:6827/2887891857] up:rejoin 2021-11-20T06:07:31.739 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:31 smithi139 conmon[33501]: cluster 2021-11-20T06:07:30.365006+0000 mon.smithi139 (mon.0) 815 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:rejoin} 1 up:standby 2021-11-20T06:07:31.739 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:31 smithi139 conmon[33501]: audit 2021-11-20T06:07:30.389152+0000 mon.smithi139 (mon.0) 816 : audit [DBG] 2021-11-20T06:07:31.740 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:31 smithi139 conmon[33501]: from='client.? 172.21.15.139:0/3728405059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:31.740 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:31 smithi139 conmon[33501]: audit 2021-11-20T06 2021-11-20T06:07:31.740 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:31 smithi139 conmon[33501]: :07:31.257950+0000 mon.smithi139 (mon.0 2021-11-20T06:07:31.740 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:31 smithi139 conmon[33501]: ) 817 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/mirror_snapshot_schedule"}]: dispatch 2021-11-20T06:07:31.740 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:31 smithi139 conmon[33501]: audit 2021- 2021-11-20T06:07:31.741 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:31 smithi139 conmon[33501]: 11-20T06:07:31.259099 2021-11-20T06:07:31.741 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:31 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 818 : 2021-11-20T06:07:31.741 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:31 smithi139 conmon[33501]: audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi139.dplyxn/trash_purge_schedule"}]: dispatch 2021-11-20T06:07:31.743 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:30.359623+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24489': {'gid': 24489, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.178:6827/2887891857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2887891857}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2887891857}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:31.743 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24489:up:rejoin 2021-11-20T06:07:32.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:32 smithi178 conmon[36600]: audit 2021-11-20T06:07:31.742591+0000 mon.smithi139 (mon.0) 819 : audit [DBG] 2021-11-20T06:07:32.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:32 smithi178 conmon[36600]: from='client.? 172.21.15.139:0/3200950516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:32.745 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:32.773 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:32 smithi139 conmon[33501]: audit 2021-11-20T06:07:31.742591+0000 mon.smithi139 (mon.0) 819 : audit [DBG] from='client.? 172.21.15.139:0/3200950516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:32.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.887+0000 7fd69c81d700 1 -- 172.21.15.139:0/1296115614 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6980f66a0 msgr2=0x7fd6980f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:32.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.887+0000 7fd69c81d700 1 --2- 172.21.15.139:0/1296115614 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6980f66a0 0x7fd6980f6ac0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fd688004660 tx=0x7fd688009b30).stop 2021-11-20T06:07:32.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.887+0000 7fd69c81d700 1 -- 172.21.15.139:0/1296115614 shutdown_connections 2021-11-20T06:07:32.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.887+0000 7fd69c81d700 1 --2- 172.21.15.139:0/1296115614 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6980f7940 0x7fd6980f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:32.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.887+0000 7fd69c81d700 1 --2- 172.21.15.139:0/1296115614 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6980f66a0 0x7fd6980f6ac0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:32.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.887+0000 7fd69c81d700 1 -- 172.21.15.139:0/1296115614 >> 172.21.15.139:0/1296115614 conn(0x7fd6980f1da0 msgr2=0x7fd6980f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:32.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.887+0000 7fd69c81d700 1 -- 172.21.15.139:0/1296115614 shutdown_connections 2021-11-20T06:07:32.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.887+0000 7fd69c81d700 1 -- 172.21.15.139:0/1296115614 wait complete. 2021-11-20T06:07:32.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd69c81d700 1 Processor -- start 2021-11-20T06:07:32.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd69c81d700 1 -- start start 2021-11-20T06:07:32.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd69c81d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6980f66a0 0x7fd698106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:32.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd69c81d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6980f7940 0x7fd698107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:32.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd69c81d700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd698107940 con 0x7fd6980f7940 2021-11-20T06:07:32.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd69c81d700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd698107a80 con 0x7fd6980f66a0 2021-11-20T06:07:32.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd695d9b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6980f7940 0x7fd698107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:32.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd695d9b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6980f7940 0x7fd698107390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58824/0 (socket says 172.21.15.139:58824) 2021-11-20T06:07:32.877 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd695d9b700 1 -- 172.21.15.139:0/1780792368 learned_addr learned my addr 172.21.15.139:0/1780792368 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:32.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd69659c700 1 --2- 172.21.15.139:0/1780792368 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6980f66a0 0x7fd698106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:32.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd695d9b700 1 -- 172.21.15.139:0/1780792368 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6980f66a0 msgr2=0x7fd698106ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:32.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd695d9b700 1 --2- 172.21.15.139:0/1780792368 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6980f66a0 0x7fd698106ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:32.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd695d9b700 1 -- 172.21.15.139:0/1780792368 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd688005040 con 0x7fd6980f7940 2021-11-20T06:07:32.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd695d9b700 1 --2- 172.21.15.139:0/1780792368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6980f7940 0x7fd698107390 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fd68c00c790 tx=0x7fd68c007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:32.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd686ffd700 1 -- 172.21.15.139:0/1780792368 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd68c00e680 con 0x7fd6980f7940 2021-11-20T06:07:32.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd686ffd700 1 -- 172.21.15.139:0/1780792368 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd68c00ec90 con 0x7fd6980f7940 2021-11-20T06:07:32.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd69c81d700 1 -- 172.21.15.139:0/1780792368 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69810c520 con 0x7fd6980f7940 2021-11-20T06:07:32.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.888+0000 7fd686ffd700 1 -- 172.21.15.139:0/1780792368 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd68c013d60 con 0x7fd6980f7940 2021-11-20T06:07:32.880 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.889+0000 7fd69c81d700 1 -- 172.21.15.139:0/1780792368 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69810cae0 con 0x7fd6980f7940 2021-11-20T06:07:32.880 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.889+0000 7fd69c81d700 1 -- 172.21.15.139:0/1780792368 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd698027a00 con 0x7fd6980f7940 2021-11-20T06:07:32.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.890+0000 7fd686ffd700 1 -- 172.21.15.139:0/1780792368 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd68c01a060 con 0x7fd6980f7940 2021-11-20T06:07:32.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.890+0000 7fd686ffd700 1 --2- 172.21.15.139:0/1780792368 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd68005c2a0 0x7fd68005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:32.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.890+0000 7fd686ffd700 1 -- 172.21.15.139:0/1780792368 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fd68c0765d0 con 0x7fd6980f7940 2021-11-20T06:07:32.882 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.890+0000 7fd69659c700 1 --2- 172.21.15.139:0/1780792368 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd68005c2a0 0x7fd68005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:32.882 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.891+0000 7fd69659c700 1 --2- 172.21.15.139:0/1780792368 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd68005c2a0 0x7fd68005e760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fd688004e30 tx=0x7fd688009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:32.882 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:32.894+0000 7fd686ffd700 1 -- 172.21.15.139:0/1780792368 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd68c048de0 con 0x7fd6980f7940 2021-11-20T06:07:33.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:33.080+0000 7fd69c81d700 1 -- 172.21.15.139:0/1780792368 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6980fee30 con 0x7fd6980f7940 2021-11-20T06:07:33.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:33.081+0000 7fd686ffd700 1 -- 172.21.15.139:0/1780792368 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2187 (secure 0 0 0) 0x7fd68c04c400 con 0x7fd6980f7940 2021-11-20T06:07:33.066 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:33.066 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:30.359623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24489":{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:33.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:33.082+0000 7fd69c81d700 1 -- 172.21.15.139:0/1780792368 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd68005c2a0 msgr2=0x7fd68005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:33.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:33.082+0000 7fd69c81d700 1 --2- 172.21.15.139:0/1780792368 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd68005c2a0 0x7fd68005e760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fd688004e30 tx=0x7fd688009b30).stop 2021-11-20T06:07:33.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:33.082+0000 7fd69c81d700 1 -- 172.21.15.139:0/1780792368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6980f7940 msgr2=0x7fd698107390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:33.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:33.082+0000 7fd69c81d700 1 --2- 172.21.15.139:0/1780792368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6980f7940 0x7fd698107390 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fd68c00c790 tx=0x7fd68c007d70).stop 2021-11-20T06:07:33.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:33.082+0000 7fd69c81d700 1 -- 172.21.15.139:0/1780792368 shutdown_connections 2021-11-20T06:07:33.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:33.082+0000 7fd69c81d700 1 --2- 172.21.15.139:0/1780792368 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd68005c2a0 0x7fd68005e760 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:33.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:33.082+0000 7fd69c81d700 1 --2- 172.21.15.139:0/1780792368 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6980f66a0 0x7fd698106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:33.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:33.082+0000 7fd69c81d700 1 --2- 172.21.15.139:0/1780792368 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd6980f7940 0x7fd698107390 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:33.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:33.082+0000 7fd69c81d700 1 -- 172.21.15.139:0/1780792368 >> 172.21.15.139:0/1780792368 conn(0x7fd6980f1da0 msgr2=0x7fd6980fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:33.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:33.082+0000 7fd69c81d700 1 -- 172.21.15.139:0/1780792368 shutdown_connections 2021-11-20T06:07:33.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:33.082+0000 7fd69c81d700 1 -- 172.21.15.139:0/1780792368 wait complete. 2021-11-20T06:07:33.070 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 77 2021-11-20T06:07:33.079 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:30.359623+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24489': {'gid': 24489, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.178:6827/2887891857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2887891857}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2887891857}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:33.079 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24489:up:rejoin 2021-11-20T06:07:33.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:33 smithi178 conmon[36600]: cluster 2021-11-20T06:07:32.272600+0000 mgr.smithi139.dplyxn (mgr.14162) 312 : cluster [DBG] pgmap v287: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:33.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:33 smithi178 conmon[36600]: cluster 2021-11-20T06:07:32.367687+0000 mon.smithi139 (mon.0) 820 : cluster [DBG] mds.? [v2:172.21.15.178:6824/1490972326,v1:172.21.15.178:6825/1490972326] up:standby 2021-11-20T06:07:33.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:33 smithi178 conmon[36600]: cluster 2021-11-20T06:07:32.367735+0000 mon.smithi139 (mon.0) 821 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:rejoin} 1 up:standby 2021-11-20T06:07:33.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:33 smithi178 conmon[36600]: audit 2021 2021-11-20T06:07:33.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:33 smithi178 conmon[36600]: -11-20T06:07:33.081590+0000 mon.smithi139 (mon.0) 822 : audit [DBG] from='client.? 172.21.15.139:0/1780792368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:33.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:33 smithi139 conmon[33501]: cluster 2021-11-20T06:07:32.272600+0000 mgr.smithi139.dplyxn (mgr.14162) 312 : cluster [DBG] pgmap v287: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:33.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:33 smithi139 conmon[33501]: cluster 2021-11-20T06:07 2021-11-20T06:07:33.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:33 smithi139 conmon[33501]: :32.367687+0000 mon.smithi139 (mon.0) 820 : cluster [DBG] mds.? [v2:172.21.15.178:6824/1490972326,v1:172.21.15.178:6825/1490972326] up:standby 2021-11-20T06:07:33.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:33 smithi139 conmon[33501]: cluster 2021-11-20T06:07:32.367735+0000 mon.smithi139 (mon.0) 821 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:rejoin} 1 up:standby 2021-11-20T06:07:33.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:33 smithi139 conmon[33501]: audit 2021-11-20T06: 2021-11-20T06:07:33.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:33 smithi139 conmon[33501]: 07:33.081590+0000 mon.smithi139 (mon.0) 822 : audit [DBG] from='client.? 172.21.15.139:0/1780792368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:34.081 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:34.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.226+0000 7f685654e700 1 -- 172.21.15.139:0/2251297506 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68500f5460 msgr2=0x7f68500f5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:34.214 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.226+0000 7f685654e700 1 --2- 172.21.15.139:0/2251297506 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68500f5460 0x7f68500f5880 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f684c004660 tx=0x7f684c009b30).stop 2021-11-20T06:07:34.214 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.227+0000 7f685654e700 1 -- 172.21.15.139:0/2251297506 shutdown_connections 2021-11-20T06:07:34.214 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.227+0000 7f685654e700 1 --2- 172.21.15.139:0/2251297506 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f68500f6700 0x7f68500f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:34.215 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.227+0000 7f685654e700 1 --2- 172.21.15.139:0/2251297506 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68500f5460 0x7f68500f5880 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:34.215 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.227+0000 7f685654e700 1 -- 172.21.15.139:0/2251297506 >> 172.21.15.139:0/2251297506 conn(0x7f68500f0ba0 msgr2=0x7f68500f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:34.215 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.227+0000 7f685654e700 1 -- 172.21.15.139:0/2251297506 shutdown_connections 2021-11-20T06:07:34.215 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.227+0000 7f685654e700 1 -- 172.21.15.139:0/2251297506 wait complete. 2021-11-20T06:07:34.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.227+0000 7f685654e700 1 Processor -- start 2021-11-20T06:07:34.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.227+0000 7f685654e700 1 -- start start 2021-11-20T06:07:34.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.227+0000 7f685654e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68500f5460 0x7f68500f95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:34.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.227+0000 7f685654e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f68500f6700 0x7f68500f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:34.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.227+0000 7f685654e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68500f7fa0 con 0x7f68500f5460 2021-11-20T06:07:34.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.228+0000 7f685654e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68500f80e0 con 0x7f68500f6700 2021-11-20T06:07:34.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.228+0000 7f685554c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68500f5460 0x7f68500f95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:34.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.228+0000 7f685554c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68500f5460 0x7f68500f95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58834/0 (socket says 172.21.15.139:58834) 2021-11-20T06:07:34.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.228+0000 7f685554c700 1 -- 172.21.15.139:0/2338110404 learned_addr learned my addr 172.21.15.139:0/2338110404 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:34.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.228+0000 7f6854d4b700 1 --2- 172.21.15.139:0/2338110404 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f68500f6700 0x7f68500f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:34.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.228+0000 7f685554c700 1 -- 172.21.15.139:0/2338110404 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f68500f6700 msgr2=0x7f68500f7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:34.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.228+0000 7f685554c700 1 --2- 172.21.15.139:0/2338110404 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f68500f6700 0x7f68500f7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:34.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.228+0000 7f685554c700 1 -- 172.21.15.139:0/2338110404 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c005040 con 0x7f68500f5460 2021-11-20T06:07:34.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.228+0000 7f685554c700 1 --2- 172.21.15.139:0/2338110404 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68500f5460 0x7f68500f95c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f684c004e30 tx=0x7f684c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:34.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.228+0000 7f6845ffb700 1 -- 172.21.15.139:0/2338110404 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f684c018070 con 0x7f68500f5460 2021-11-20T06:07:34.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.228+0000 7f685654e700 1 -- 172.21.15.139:0/2338110404 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68500f8360 con 0x7f68500f5460 2021-11-20T06:07:34.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.228+0000 7f6845ffb700 1 -- 172.21.15.139:0/2338110404 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f684c013410 con 0x7f68500f5460 2021-11-20T06:07:34.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.228+0000 7f6845ffb700 1 -- 172.21.15.139:0/2338110404 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f684c01de20 con 0x7f68500f5460 2021-11-20T06:07:34.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.228+0000 7f685654e700 1 -- 172.21.15.139:0/2338110404 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68500f88c0 con 0x7f68500f5460 2021-11-20T06:07:34.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.229+0000 7f685654e700 1 -- 172.21.15.139:0/2338110404 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6850045ad0 con 0x7f68500f5460 2021-11-20T06:07:34.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.230+0000 7f6845ffb700 1 -- 172.21.15.139:0/2338110404 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f684c01c4a0 con 0x7f68500f5460 2021-11-20T06:07:34.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.230+0000 7f6845ffb700 1 --2- 172.21.15.139:0/2338110404 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f683c05c300 0x7f683c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:34.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.230+0000 7f6845ffb700 1 -- 172.21.15.139:0/2338110404 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f684c077fb0 con 0x7f68500f5460 2021-11-20T06:07:34.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.230+0000 7f6854d4b700 1 --2- 172.21.15.139:0/2338110404 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f683c05c300 0x7f683c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:34.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.230+0000 7f6854d4b700 1 --2- 172.21.15.139:0/2338110404 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f683c05c300 0x7f683c05e7c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f6840004d80 tx=0x7f684000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:34.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.234+0000 7f6845ffb700 1 -- 172.21.15.139:0/2338110404 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f684c04a7c0 con 0x7f68500f5460 2021-11-20T06:07:34.404 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.420+0000 7f685654e700 1 -- 172.21.15.139:0/2338110404 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68500fdbf0 con 0x7f68500f5460 2021-11-20T06:07:34.405 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.420+0000 7f6845ffb700 1 -- 172.21.15.139:0/2338110404 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f684c04dde0 con 0x7f68500f5460 2021-11-20T06:07:34.405 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:34.406 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:30.359623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24489":{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:34.407 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.422+0000 7f685654e700 1 -- 172.21.15.139:0/2338110404 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f683c05c300 msgr2=0x7f683c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:34.407 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.422+0000 7f685654e700 1 --2- 172.21.15.139:0/2338110404 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f683c05c300 0x7f683c05e7c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f6840004d80 tx=0x7f684000b040).stop 2021-11-20T06:07:34.407 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.422+0000 7f685654e700 1 -- 172.21.15.139:0/2338110404 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68500f5460 msgr2=0x7f68500f95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:34.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.422+0000 7f685654e700 1 --2- 172.21.15.139:0/2338110404 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68500f5460 0x7f68500f95c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f684c004e30 tx=0x7f684c009b30).stop 2021-11-20T06:07:34.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.422+0000 7f685654e700 1 -- 172.21.15.139:0/2338110404 shutdown_connections 2021-11-20T06:07:34.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.422+0000 7f685654e700 1 --2- 172.21.15.139:0/2338110404 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f683c05c300 0x7f683c05e7c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:34.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.422+0000 7f685654e700 1 --2- 172.21.15.139:0/2338110404 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f68500f5460 0x7f68500f95c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:34.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.422+0000 7f685654e700 1 --2- 172.21.15.139:0/2338110404 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f68500f6700 0x7f68500f7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:34.409 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.422+0000 7f685654e700 1 -- 172.21.15.139:0/2338110404 >> 172.21.15.139:0/2338110404 conn(0x7f68500f0ba0 msgr2=0x7f685005fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:34.409 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.422+0000 7f685654e700 1 -- 172.21.15.139:0/2338110404 shutdown_connections 2021-11-20T06:07:34.409 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:34.422+0000 7f685654e700 1 -- 172.21.15.139:0/2338110404 wait complete. 2021-11-20T06:07:34.409 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 77 2021-11-20T06:07:34.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:30.359623+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24489': {'gid': 24489, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.178:6827/2887891857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2887891857}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2887891857}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:34.419 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24489:up:rejoin 2021-11-20T06:07:35.420 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:35.447 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:35 smithi139 conmon[33501]: cluster 2021-11-20T06:07:34.273017+0000 mgr.smithi139.dplyxn (mgr.14162) 313 : cluster [DBG] pgmap v288: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:35.447 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:35 smithi139 conmon[33501]: 2021-11-20T06:07:35.447 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:35 smithi139 conmon[33501]: audit 2021-11-20T06:07:34.420994+0000 mon.smithi139 (mon.0) 823 : audit [DBG] from='client.? 172.21.15.139:0/2338110404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:35.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.561+0000 7fec5a3e3700 1 -- 172.21.15.139:0/576155487 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec540f7510 msgr2=0x7fec540f99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:35.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.561+0000 7fec5a3e3700 1 --2- 172.21.15.139:0/576155487 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec540f7510 0x7fec540f99a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fec44004660 tx=0x7fec44009b30).stop 2021-11-20T06:07:35.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.561+0000 7fec5a3e3700 1 -- 172.21.15.139:0/576155487 shutdown_connections 2021-11-20T06:07:35.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.561+0000 7fec5a3e3700 1 --2- 172.21.15.139:0/576155487 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec540f7510 0x7fec540f99a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:35.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.561+0000 7fec5a3e3700 1 --2- 172.21.15.139:0/576155487 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec54060490 0x7fec540f6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:35.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.562+0000 7fec5a3e3700 1 -- 172.21.15.139:0/576155487 >> 172.21.15.139:0/576155487 conn(0x7fec540f0c40 msgr2=0x7fec540f3060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:35.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.562+0000 7fec5a3e3700 1 -- 172.21.15.139:0/576155487 shutdown_connections 2021-11-20T06:07:35.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.562+0000 7fec5a3e3700 1 -- 172.21.15.139:0/576155487 wait complete. 2021-11-20T06:07:35.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.562+0000 7fec5a3e3700 1 Processor -- start 2021-11-20T06:07:35.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.562+0000 7fec5a3e3700 1 -- start start 2021-11-20T06:07:35.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.562+0000 7fec5a3e3700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec54060490 0x7fec5410d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:35.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.562+0000 7fec5a3e3700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec540f7510 0x7fec5410d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:35.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.562+0000 7fec5a3e3700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5410dd90 con 0x7fec54060490 2021-11-20T06:07:35.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.562+0000 7fec5a3e3700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5410ded0 con 0x7fec540f7510 2021-11-20T06:07:35.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.562+0000 7fec593e1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec54060490 0x7fec5410d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:35.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.562+0000 7fec593e1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec54060490 0x7fec5410d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58846/0 (socket says 172.21.15.139:58846) 2021-11-20T06:07:35.556 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.562+0000 7fec593e1700 1 -- 172.21.15.139:0/3459692083 learned_addr learned my addr 172.21.15.139:0/3459692083 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:35.556 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.563+0000 7fec58be0700 1 --2- 172.21.15.139:0/3459692083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec540f7510 0x7fec5410d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:35.556 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.563+0000 7fec593e1700 1 -- 172.21.15.139:0/3459692083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec540f7510 msgr2=0x7fec5410d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:35.556 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.563+0000 7fec593e1700 1 --2- 172.21.15.139:0/3459692083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec540f7510 0x7fec5410d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:35.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.563+0000 7fec593e1700 1 -- 172.21.15.139:0/3459692083 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec44005040 con 0x7fec54060490 2021-11-20T06:07:35.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.563+0000 7fec593e1700 1 --2- 172.21.15.139:0/3459692083 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec54060490 0x7fec5410d2f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fec500026d0 tx=0x7fec50009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:35.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.563+0000 7fec49ffb700 1 -- 172.21.15.139:0/3459692083 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec500079c0 con 0x7fec54060490 2021-11-20T06:07:35.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.563+0000 7fec49ffb700 1 -- 172.21.15.139:0/3459692083 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec5000f040 con 0x7fec54060490 2021-11-20T06:07:35.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.563+0000 7fec5a3e3700 1 -- 172.21.15.139:0/3459692083 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5410e870 con 0x7fec54060490 2021-11-20T06:07:35.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.563+0000 7fec49ffb700 1 -- 172.21.15.139:0/3459692083 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec500148b0 con 0x7fec54060490 2021-11-20T06:07:35.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.563+0000 7fec5a3e3700 1 -- 172.21.15.139:0/3459692083 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5410ee30 con 0x7fec54060490 2021-11-20T06:07:35.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.564+0000 7fec5a3e3700 1 -- 172.21.15.139:0/3459692083 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec54045ad0 con 0x7fec54060490 2021-11-20T06:07:35.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.564+0000 7fec49ffb700 1 -- 172.21.15.139:0/3459692083 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fec50007b20 con 0x7fec54060490 2021-11-20T06:07:35.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.565+0000 7fec49ffb700 1 --2- 172.21.15.139:0/3459692083 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fec4005c300 0x7fec4005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:35.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.565+0000 7fec49ffb700 1 -- 172.21.15.139:0/3459692083 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fec500762f0 con 0x7fec54060490 2021-11-20T06:07:35.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.565+0000 7fec58be0700 1 --2- 172.21.15.139:0/3459692083 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fec4005c300 0x7fec4005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:35.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.565+0000 7fec58be0700 1 --2- 172.21.15.139:0/3459692083 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fec4005c300 0x7fec4005e7c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fec44009f50 tx=0x7fec44009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:35.560 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.569+0000 7fec49ffb700 1 -- 172.21.15.139:0/3459692083 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec50048b00 con 0x7fec54060490 2021-11-20T06:07:35.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:35 smithi178 conmon[36600]: cluster 2021-11-20T06:07:34.273017+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:07:35.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:35 smithi178 conmon[36600]: 313 : cluster [DBG] pgmap v288: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:35.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:35 smithi178 conmon[36600]: audit 2021-11-20T06:07:34.420994+0000 mon.smithi139 (mon.0) 823 : audit [DBG] from='client.? 172.21.15.139:0/2338110404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:35.741 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.757+0000 7fec5a3e3700 1 -- 172.21.15.139:0/3459692083 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec54027a00 con 0x7fec54060490 2021-11-20T06:07:35.742 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.758+0000 7fec49ffb700 1 -- 172.21.15.139:0/3459692083 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2187 (secure 0 0 0) 0x7fec5004c120 con 0x7fec54060490 2021-11-20T06:07:35.743 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:35.743 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:30.359623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24489":{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:35.744 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.759+0000 7fec5a3e3700 1 -- 172.21.15.139:0/3459692083 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fec4005c300 msgr2=0x7fec4005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:35.744 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.759+0000 7fec5a3e3700 1 --2- 172.21.15.139:0/3459692083 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fec4005c300 0x7fec4005e7c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fec44009f50 tx=0x7fec44009b30).stop 2021-11-20T06:07:35.745 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.759+0000 7fec5a3e3700 1 -- 172.21.15.139:0/3459692083 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec54060490 msgr2=0x7fec5410d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:35.745 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.759+0000 7fec5a3e3700 1 --2- 172.21.15.139:0/3459692083 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec54060490 0x7fec5410d2f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fec500026d0 tx=0x7fec50009de0).stop 2021-11-20T06:07:35.745 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.759+0000 7fec5a3e3700 1 -- 172.21.15.139:0/3459692083 shutdown_connections 2021-11-20T06:07:35.745 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.759+0000 7fec5a3e3700 1 --2- 172.21.15.139:0/3459692083 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fec4005c300 0x7fec4005e7c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:35.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.759+0000 7fec5a3e3700 1 --2- 172.21.15.139:0/3459692083 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fec54060490 0x7fec5410d2f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:35.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.759+0000 7fec5a3e3700 1 --2- 172.21.15.139:0/3459692083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec540f7510 0x7fec5410d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:35.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.759+0000 7fec5a3e3700 1 -- 172.21.15.139:0/3459692083 >> 172.21.15.139:0/3459692083 conn(0x7fec540f0c40 msgr2=0x7fec540f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:35.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.759+0000 7fec5a3e3700 1 -- 172.21.15.139:0/3459692083 shutdown_connections 2021-11-20T06:07:35.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:35.759+0000 7fec5a3e3700 1 -- 172.21.15.139:0/3459692083 wait complete. 2021-11-20T06:07:35.747 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 77 2021-11-20T06:07:35.757 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:30.359623+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24489': {'gid': 24489, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.178:6827/2887891857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2887891857}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2887891857}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:35.757 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24489:up:rejoin 2021-11-20T06:07:36.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:36 smithi178 conmon[36600]: audit 2021-11-20T06:07:35.758478+0000 mon.smithi139 (mon.0) 824 : audit [DBG] from='client.? 172.21.15.139:0/3459692083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:36.759 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:36.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:36 smithi139 conmon[33501]: audit 2021-11-20T06:07:35.758478+0000 mon.smithi139 (mon.0) 824 2021-11-20T06:07:36.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:36 smithi139 conmon[33501]: : audit [DBG] from='client.? 172.21.15.139:0/3459692083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:36.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.901+0000 7f8932182700 1 -- 172.21.15.139:0/3282931410 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f892c0f7870 msgr2=0x7f892c0f7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:36.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.901+0000 7f8932182700 1 --2- 172.21.15.139:0/3282931410 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f892c0f7870 0x7f892c0f7cd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f891c004660 tx=0x7f891c009b30).stop 2021-11-20T06:07:36.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.902+0000 7f8932182700 1 -- 172.21.15.139:0/3282931410 shutdown_connections 2021-11-20T06:07:36.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.902+0000 7f8932182700 1 --2- 172.21.15.139:0/3282931410 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f892c0f7870 0x7f892c0f7cd0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:36.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.902+0000 7f8932182700 1 --2- 172.21.15.139:0/3282931410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f892c0f65d0 0x7f892c0f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:36.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.902+0000 7f8932182700 1 -- 172.21.15.139:0/3282931410 >> 172.21.15.139:0/3282931410 conn(0x7f892c0f1d10 msgr2=0x7f892c0f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:36.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.902+0000 7f8932182700 1 -- 172.21.15.139:0/3282931410 shutdown_connections 2021-11-20T06:07:36.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.902+0000 7f8932182700 1 -- 172.21.15.139:0/3282931410 wait complete. 2021-11-20T06:07:36.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.902+0000 7f8932182700 1 Processor -- start 2021-11-20T06:07:36.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.902+0000 7f8932182700 1 -- start start 2021-11-20T06:07:36.890 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.902+0000 7f8932182700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f892c0f65d0 0x7f892c106d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:36.890 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.902+0000 7f8932182700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f892c0f7870 0x7f892c107280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:36.890 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.902+0000 7f8932182700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c107880 con 0x7f892c0f7870 2021-11-20T06:07:36.890 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.902+0000 7f8932182700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c108090 con 0x7f892c0f65d0 2021-11-20T06:07:36.891 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.902+0000 7f893097f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f892c0f7870 0x7f892c107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:36.891 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.902+0000 7f893097f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f892c0f7870 0x7f892c107280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58858/0 (socket says 172.21.15.139:58858) 2021-11-20T06:07:36.891 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.902+0000 7f893097f700 1 -- 172.21.15.139:0/1090937741 learned_addr learned my addr 172.21.15.139:0/1090937741 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:36.893 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.903+0000 7f8931180700 1 --2- 172.21.15.139:0/1090937741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f892c0f65d0 0x7f892c106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:36.894 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.903+0000 7f893097f700 1 -- 172.21.15.139:0/1090937741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f892c0f65d0 msgr2=0x7f892c106d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:36.894 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.903+0000 7f893097f700 1 --2- 172.21.15.139:0/1090937741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f892c0f65d0 0x7f892c106d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:36.894 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.903+0000 7f893097f700 1 -- 172.21.15.139:0/1090937741 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c005040 con 0x7f892c0f7870 2021-11-20T06:07:36.894 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.903+0000 7f893097f700 1 --2- 172.21.15.139:0/1090937741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f892c0f7870 0x7f892c107280 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f891c009f50 tx=0x7f891c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:36.895 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.903+0000 7f8921ffb700 1 -- 172.21.15.139:0/1090937741 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f891c018070 con 0x7f892c0f7870 2021-11-20T06:07:36.895 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.903+0000 7f8921ffb700 1 -- 172.21.15.139:0/1090937741 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f891c013410 con 0x7f892c0f7870 2021-11-20T06:07:36.895 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.903+0000 7f8932182700 1 -- 172.21.15.139:0/1090937741 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892c1082b0 con 0x7f892c0f7870 2021-11-20T06:07:36.895 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.903+0000 7f8921ffb700 1 -- 172.21.15.139:0/1090937741 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f891c01c3c0 con 0x7f892c0f7870 2021-11-20T06:07:36.895 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.903+0000 7f8932182700 1 -- 172.21.15.139:0/1090937741 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f892c108810 con 0x7f892c0f7870 2021-11-20T06:07:36.896 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.904+0000 7f8932182700 1 -- 172.21.15.139:0/1090937741 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892c0f59c0 con 0x7f892c0f7870 2021-11-20T06:07:36.896 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.905+0000 7f8921ffb700 1 -- 172.21.15.139:0/1090937741 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f891c01d420 con 0x7f892c0f7870 2021-11-20T06:07:36.896 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.905+0000 7f8921ffb700 1 --2- 172.21.15.139:0/1090937741 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f891805c300 0x7f891805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:36.896 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.905+0000 7f8921ffb700 1 -- 172.21.15.139:0/1090937741 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f891c077ff0 con 0x7f892c0f7870 2021-11-20T06:07:36.897 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.905+0000 7f8931180700 1 --2- 172.21.15.139:0/1090937741 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f891805c300 0x7f891805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:36.897 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.905+0000 7f8931180700 1 --2- 172.21.15.139:0/1090937741 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f891805c300 0x7f891805e7c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f892c106b00 tx=0x7f892800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:36.897 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:36.909+0000 7f8921ffb700 1 -- 172.21.15.139:0/1090937741 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f891c04a800 con 0x7f892c0f7870 2021-11-20T06:07:37.076 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:37.092+0000 7f8932182700 1 -- 172.21.15.139:0/1090937741 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f892c0fee80 con 0x7f892c0f7870 2021-11-20T06:07:37.077 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:37.092+0000 7f8921ffb700 1 -- 172.21.15.139:0/1090937741 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f892c0fee80 con 0x7f892c0f7870 2021-11-20T06:07:37.077 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:37.078 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:30.359623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24489":{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:37.079 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:37.094+0000 7f8932182700 1 -- 172.21.15.139:0/1090937741 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f891805c300 msgr2=0x7f891805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:37.079 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:37.094+0000 7f8932182700 1 --2- 172.21.15.139:0/1090937741 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f891805c300 0x7f891805e7c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f892c106b00 tx=0x7f892800b040).stop 2021-11-20T06:07:37.080 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:37.094+0000 7f8932182700 1 -- 172.21.15.139:0/1090937741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f892c0f7870 msgr2=0x7f892c107280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:37.080 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:37.094+0000 7f8932182700 1 --2- 172.21.15.139:0/1090937741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f892c0f7870 0x7f892c107280 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f891c009f50 tx=0x7f891c009b30).stop 2021-11-20T06:07:37.080 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:37.094+0000 7f8932182700 1 -- 172.21.15.139:0/1090937741 shutdown_connections 2021-11-20T06:07:37.080 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:37.094+0000 7f8932182700 1 --2- 172.21.15.139:0/1090937741 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f891805c300 0x7f891805e7c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:37.081 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:37.094+0000 7f8932182700 1 --2- 172.21.15.139:0/1090937741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f892c0f65d0 0x7f892c106d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:37.081 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:37.094+0000 7f8932182700 1 --2- 172.21.15.139:0/1090937741 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f892c0f7870 0x7f892c107280 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:37.081 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:37.094+0000 7f8932182700 1 -- 172.21.15.139:0/1090937741 >> 172.21.15.139:0/1090937741 conn(0x7f892c0f1d10 msgr2=0x7f892c0fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:37.081 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:37.094+0000 7f8932182700 1 -- 172.21.15.139:0/1090937741 shutdown_connections 2021-11-20T06:07:37.082 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:37.094+0000 7f8932182700 1 -- 172.21.15.139:0/1090937741 wait complete. 2021-11-20T06:07:37.082 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 77 2021-11-20T06:07:37.090 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:30.359623+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24489': {'gid': 24489, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.178:6827/2887891857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2887891857}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2887891857}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:37.091 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24489:up:rejoin 2021-11-20T06:07:37.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:37 smithi178 conmon[36600]: cluster 2021-11-20T06:07:36.273677+0000 mgr.smithi139.dplyxn (mgr. 2021-11-20T06:07:37.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:37 smithi178 conmon[36600]: 14162) 314 : cluster [DBG] pgmap v289: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:37.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:37 smithi178 conmon[36600]: audit 2021-11-20T06:07:37.093135+0000 mon.smithi139 (mon.0) 825 : audit [DBG] from='client.? 172.21.15.139:0/1090937741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:37.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:37 smithi178 conmon[36600]: 2021-11-20T06:07:37.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:37 smithi139 conmon[33501]: cluster 2021-11-20T06:07:36.273677+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:07:37.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:37 smithi139 conmon[33501]: mgr.14162) 314 : cluster [DBG] pgmap v289: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:37.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:37 smithi139 conmon[33501]: audit 2021-11-20T06:07:37.093135+0000 mon.smithi139 (mon.0) 825 : audit [DBG] from='client.? 172.21.15.139:0/1090937741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:38.092 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:38.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.234+0000 7fa999078700 1 -- 172.21.15.139:0/715828885 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9940f7940 msgr2=0x7fa9940f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:38.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.234+0000 7fa999078700 1 --2- 172.21.15.139:0/715828885 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9940f7940 0x7fa9940f7da0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fa988004660 tx=0x7fa988009b30).stop 2021-11-20T06:07:38.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.234+0000 7fa999078700 1 -- 172.21.15.139:0/715828885 shutdown_connections 2021-11-20T06:07:38.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.234+0000 7fa999078700 1 --2- 172.21.15.139:0/715828885 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9940f7940 0x7fa9940f7da0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:38.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.234+0000 7fa999078700 1 --2- 172.21.15.139:0/715828885 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9940f66a0 0x7fa9940f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:38.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.234+0000 7fa999078700 1 -- 172.21.15.139:0/715828885 >> 172.21.15.139:0/715828885 conn(0x7fa9940f1da0 msgr2=0x7fa9940f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:38.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.234+0000 7fa999078700 1 -- 172.21.15.139:0/715828885 shutdown_connections 2021-11-20T06:07:38.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.234+0000 7fa999078700 1 -- 172.21.15.139:0/715828885 wait complete. 2021-11-20T06:07:38.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.234+0000 7fa999078700 1 Processor -- start 2021-11-20T06:07:38.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.234+0000 7fa999078700 1 -- start start 2021-11-20T06:07:38.223 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa999078700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9940f66a0 0x7fa99406b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:38.223 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa999078700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9940f7940 0x7fa994069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:38.223 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa999078700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99406bd70 con 0x7fa9940f66a0 2021-11-20T06:07:38.223 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa999078700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99406a310 con 0x7fa9940f7940 2021-11-20T06:07:38.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa992d9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9940f66a0 0x7fa99406b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:38.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa992d9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9940f66a0 0x7fa99406b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58866/0 (socket says 172.21.15.139:58866) 2021-11-20T06:07:38.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa992d9d700 1 -- 172.21.15.139:0/1617709012 learned_addr learned my addr 172.21.15.139:0/1617709012 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:38.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa99259c700 1 --2- 172.21.15.139:0/1617709012 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9940f7940 0x7fa994069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:38.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa992d9d700 1 -- 172.21.15.139:0/1617709012 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9940f7940 msgr2=0x7fa994069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:38.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa992d9d700 1 --2- 172.21.15.139:0/1617709012 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9940f7940 0x7fa994069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:38.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa992d9d700 1 -- 172.21.15.139:0/1617709012 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa988005040 con 0x7fa9940f66a0 2021-11-20T06:07:38.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa992d9d700 1 --2- 172.21.15.139:0/1617709012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9940f66a0 0x7fa99406b7c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fa984002700 tx=0x7fa984009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:38.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa97b7fe700 1 -- 172.21.15.139:0/1617709012 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9840079c0 con 0x7fa9940f66a0 2021-11-20T06:07:38.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa97b7fe700 1 -- 172.21.15.139:0/1617709012 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa98400f040 con 0x7fa9940f66a0 2021-11-20T06:07:38.226 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa999078700 1 -- 172.21.15.139:0/1617709012 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99406a590 con 0x7fa9940f66a0 2021-11-20T06:07:38.226 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa97b7fe700 1 -- 172.21.15.139:0/1617709012 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa984014860 con 0x7fa9940f66a0 2021-11-20T06:07:38.226 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.235+0000 7fa999078700 1 -- 172.21.15.139:0/1617709012 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99406ab50 con 0x7fa9940f66a0 2021-11-20T06:07:38.226 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.236+0000 7fa999078700 1 -- 172.21.15.139:0/1617709012 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa994027a00 con 0x7fa9940f66a0 2021-11-20T06:07:38.227 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.236+0000 7fa97b7fe700 1 -- 172.21.15.139:0/1617709012 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa98401b060 con 0x7fa9940f66a0 2021-11-20T06:07:38.228 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.237+0000 7fa97b7fe700 1 --2- 172.21.15.139:0/1617709012 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa97c05c2a0 0x7fa97c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:38.228 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.237+0000 7fa97b7fe700 1 -- 172.21.15.139:0/1617709012 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa984076260 con 0x7fa9940f66a0 2021-11-20T06:07:38.228 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.237+0000 7fa99259c700 1 --2- 172.21.15.139:0/1617709012 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa97c05c2a0 0x7fa97c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:38.228 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.237+0000 7fa99259c700 1 --2- 172.21.15.139:0/1617709012 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa97c05c2a0 0x7fa97c05e760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fa988009f50 tx=0x7fa988009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:38.229 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.241+0000 7fa97b7fe700 1 -- 172.21.15.139:0/1617709012 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa984048a70 con 0x7fa9940f66a0 2021-11-20T06:07:38.411 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.427+0000 7fa999078700 1 -- 172.21.15.139:0/1617709012 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9940fee30 con 0x7fa9940f66a0 2021-11-20T06:07:38.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.427+0000 7fa97b7fe700 1 -- 172.21.15.139:0/1617709012 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2187 (secure 0 0 0) 0x7fa98404c090 con 0x7fa9940f66a0 2021-11-20T06:07:38.412 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:38.413 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:30.359623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24489":{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:38.414 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.429+0000 7fa999078700 1 -- 172.21.15.139:0/1617709012 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa97c05c2a0 msgr2=0x7fa97c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:38.415 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.429+0000 7fa999078700 1 --2- 172.21.15.139:0/1617709012 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa97c05c2a0 0x7fa97c05e760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fa988009f50 tx=0x7fa988009b30).stop 2021-11-20T06:07:38.415 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.429+0000 7fa999078700 1 -- 172.21.15.139:0/1617709012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9940f66a0 msgr2=0x7fa99406b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:38.415 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.429+0000 7fa999078700 1 --2- 172.21.15.139:0/1617709012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9940f66a0 0x7fa99406b7c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fa984002700 tx=0x7fa984009d80).stop 2021-11-20T06:07:38.415 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.429+0000 7fa999078700 1 -- 172.21.15.139:0/1617709012 shutdown_connections 2021-11-20T06:07:38.416 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.429+0000 7fa999078700 1 --2- 172.21.15.139:0/1617709012 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa97c05c2a0 0x7fa97c05e760 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:38.416 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.429+0000 7fa999078700 1 --2- 172.21.15.139:0/1617709012 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9940f66a0 0x7fa99406b7c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:38.416 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.429+0000 7fa999078700 1 --2- 172.21.15.139:0/1617709012 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9940f7940 0x7fa994069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:38.416 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.429+0000 7fa999078700 1 -- 172.21.15.139:0/1617709012 >> 172.21.15.139:0/1617709012 conn(0x7fa9940f1da0 msgr2=0x7fa9940fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:38.417 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.429+0000 7fa999078700 1 -- 172.21.15.139:0/1617709012 shutdown_connections 2021-11-20T06:07:38.417 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:38.429+0000 7fa999078700 1 -- 172.21.15.139:0/1617709012 wait complete. 2021-11-20T06:07:38.417 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 77 2021-11-20T06:07:38.426 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:30.359623+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24489': {'gid': 24489, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.178:6827/2887891857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2887891857}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2887891857}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:38.426 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24489:up:rejoin 2021-11-20T06:07:39.427 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:39.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:39 smithi139 conmon[33501]: cluster 2021-11-20T06:07:38.274054+0000 mgr.smithi139.dplyxn (mgr.14162) 315 : cluster [DBG] pgmap v290: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:39.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:39 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:07:39.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:39 smithi139 conmon[33501]: 20T06:07:38.428029+0000 mon.smithi139 (mon.0) 826 : audit [DBG] from='client.? 172.21.15.139:0/1617709012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:39.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.557+0000 7f65d5014700 1 -- 172.21.15.139:0/2188393587 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65d00f6bb0 msgr2=0x7f65d00f6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:39.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.557+0000 7f65d5014700 1 --2- 172.21.15.139:0/2188393587 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65d00f6bb0 0x7f65d00f6fd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f65c0004660 tx=0x7f65c0009b30).stop 2021-11-20T06:07:39.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.557+0000 7f65d5014700 1 -- 172.21.15.139:0/2188393587 shutdown_connections 2021-11-20T06:07:39.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.557+0000 7f65d5014700 1 --2- 172.21.15.139:0/2188393587 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65d00f7550 0x7f65d00f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:39.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.557+0000 7f65d5014700 1 --2- 172.21.15.139:0/2188393587 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65d00f6bb0 0x7f65d00f6fd0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:39.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.557+0000 7f65d5014700 1 -- 172.21.15.139:0/2188393587 >> 172.21.15.139:0/2188393587 conn(0x7f65d00f0440 msgr2=0x7f65d00f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:39.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.557+0000 7f65d5014700 1 -- 172.21.15.139:0/2188393587 shutdown_connections 2021-11-20T06:07:39.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.557+0000 7f65d5014700 1 -- 172.21.15.139:0/2188393587 wait complete. 2021-11-20T06:07:39.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.558+0000 7f65d5014700 1 Processor -- start 2021-11-20T06:07:39.548 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.558+0000 7f65d5014700 1 -- start start 2021-11-20T06:07:39.548 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.558+0000 7f65d5014700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65d00f6bb0 0x7f65d00f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:39.548 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.558+0000 7f65d5014700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65d00f7550 0x7f65d010bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:39.549 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.558+0000 7f65d5014700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d010c1e0 con 0x7f65d00f6bb0 2021-11-20T06:07:39.549 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.558+0000 7f65d5014700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d010c320 con 0x7f65d00f7550 2021-11-20T06:07:39.549 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.558+0000 7f65ced9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65d00f6bb0 0x7f65d00f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:39.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.558+0000 7f65ced9d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65d00f6bb0 0x7f65d00f5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58878/0 (socket says 172.21.15.139:58878) 2021-11-20T06:07:39.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.558+0000 7f65ced9d700 1 -- 172.21.15.139:0/2457558725 learned_addr learned my addr 172.21.15.139:0/2457558725 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:39.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.558+0000 7f65ce59c700 1 --2- 172.21.15.139:0/2457558725 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65d00f7550 0x7f65d010bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:39.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.559+0000 7f65ced9d700 1 -- 172.21.15.139:0/2457558725 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65d00f7550 msgr2=0x7f65d010bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:39.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.559+0000 7f65ced9d700 1 --2- 172.21.15.139:0/2457558725 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65d00f7550 0x7f65d010bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:39.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.559+0000 7f65ced9d700 1 -- 172.21.15.139:0/2457558725 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c0005040 con 0x7f65d00f6bb0 2021-11-20T06:07:39.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.559+0000 7f65ce59c700 1 --2- 172.21.15.139:0/2457558725 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65d00f7550 0x7f65d010bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:07:39.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.559+0000 7f65ced9d700 1 --2- 172.21.15.139:0/2457558725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65d00f6bb0 0x7f65d00f5310 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f65c0009830 tx=0x7f65c0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:39.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.559+0000 7f65b77fe700 1 -- 172.21.15.139:0/2457558725 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65c0018070 con 0x7f65d00f6bb0 2021-11-20T06:07:39.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.559+0000 7f65b77fe700 1 -- 172.21.15.139:0/2457558725 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65c0013410 con 0x7f65d00f6bb0 2021-11-20T06:07:39.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.559+0000 7f65d5014700 1 -- 172.21.15.139:0/2457558725 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d010c520 con 0x7f65d00f6bb0 2021-11-20T06:07:39.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.559+0000 7f65b77fe700 1 -- 172.21.15.139:0/2457558725 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65c001c620 con 0x7f65d00f6bb0 2021-11-20T06:07:39.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.559+0000 7f65d5014700 1 -- 172.21.15.139:0/2457558725 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d010ca80 con 0x7f65d00f6bb0 2021-11-20T06:07:39.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.560+0000 7f65d5014700 1 -- 172.21.15.139:0/2457558725 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d0027a00 con 0x7f65d00f6bb0 2021-11-20T06:07:39.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.560+0000 7f65b77fe700 1 -- 172.21.15.139:0/2457558725 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f65c001d420 con 0x7f65d00f6bb0 2021-11-20T06:07:39.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.560+0000 7f65b77fe700 1 --2- 172.21.15.139:0/2457558725 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f65b805c2f0 0x7f65b805e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:39.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.560+0000 7f65b77fe700 1 -- 172.21.15.139:0/2457558725 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f65c0078270 con 0x7f65d00f6bb0 2021-11-20T06:07:39.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.561+0000 7f65ce59c700 1 --2- 172.21.15.139:0/2457558725 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f65b805c2f0 0x7f65b805e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:39.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.561+0000 7f65ce59c700 1 --2- 172.21.15.139:0/2457558725 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f65b805c2f0 0x7f65b805e7b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f65c40041d0 tx=0x7f65c4009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:39.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.564+0000 7f65b77fe700 1 -- 172.21.15.139:0/2457558725 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f65c004aa40 con 0x7f65d00f6bb0 2021-11-20T06:07:39.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:39 smithi178 conmon[36600]: cluster 2021-11-20T06:07:38.274054+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:07:39.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:39 smithi178 conmon[36600]: 315 : cluster [DBG] pgmap v290: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:39.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:39 smithi178 conmon[36600]: audit 2021-11-20T06:07:38.428029+0000 mon.smithi139 (mon.0) 826 : audit [DBG] from='client.? 172.21.15.139:0/1617709012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:39.733 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.749+0000 7f65d5014700 1 -- 172.21.15.139:0/2457558725 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65d00fdd40 con 0x7f65d00f6bb0 2021-11-20T06:07:39.734 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.749+0000 7f65b77fe700 1 -- 172.21.15.139:0/2457558725 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f65c004e060 con 0x7f65d00f6bb0 2021-11-20T06:07:39.735 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:39.735 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:30.359623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24489":{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:39.736 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.751+0000 7f65d5014700 1 -- 172.21.15.139:0/2457558725 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f65b805c2f0 msgr2=0x7f65b805e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:39.736 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.751+0000 7f65d5014700 1 --2- 172.21.15.139:0/2457558725 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f65b805c2f0 0x7f65b805e7b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f65c40041d0 tx=0x7f65c4009040).stop 2021-11-20T06:07:39.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.751+0000 7f65d5014700 1 -- 172.21.15.139:0/2457558725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65d00f6bb0 msgr2=0x7f65d00f5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:39.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.751+0000 7f65d5014700 1 --2- 172.21.15.139:0/2457558725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65d00f6bb0 0x7f65d00f5310 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f65c0009830 tx=0x7f65c0009b30).stop 2021-11-20T06:07:39.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.751+0000 7f65d5014700 1 -- 172.21.15.139:0/2457558725 shutdown_connections 2021-11-20T06:07:39.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.751+0000 7f65d5014700 1 --2- 172.21.15.139:0/2457558725 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f65b805c2f0 0x7f65b805e7b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:39.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.751+0000 7f65d5014700 1 --2- 172.21.15.139:0/2457558725 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f65d00f6bb0 0x7f65d00f5310 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:39.738 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.751+0000 7f65d5014700 1 --2- 172.21.15.139:0/2457558725 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65d00f7550 0x7f65d010bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:39.738 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.751+0000 7f65d5014700 1 -- 172.21.15.139:0/2457558725 >> 172.21.15.139:0/2457558725 conn(0x7f65d00f0440 msgr2=0x7f65d00fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:39.738 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.751+0000 7f65d5014700 1 -- 172.21.15.139:0/2457558725 shutdown_connections 2021-11-20T06:07:39.738 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:39.751+0000 7f65d5014700 1 -- 172.21.15.139:0/2457558725 wait complete. 2021-11-20T06:07:39.739 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 77 2021-11-20T06:07:39.748 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:30.359623+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24489': {'gid': 24489, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.178:6827/2887891857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2887891857}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2887891857}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:39.748 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24489:up:rejoin 2021-11-20T06:07:40.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:40 smithi178 conmon[36600]: audit 2021-11-20T06:07:39.750345+0000 mon.smithi139 (mon.0) 827 2021-11-20T06:07:40.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:40 smithi178 conmon[36600]: : audit [DBG] from='client.? 172.21.15.139:0/2457558725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:40.750 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:40.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:40 smithi139 conmon[33501]: audit 2021-11-20T06:07:39.750345+0000 mon.smithi139 (mon.0) 827 : audit [DBG] from='client.? 172.21.15.139:0/2457558725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:40.885 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.893+0000 7f59d674b700 1 -- 172.21.15.139:0/3068517571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59d00f5580 msgr2=0x7f59d00f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:40.886 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.893+0000 7f59d674b700 1 --2- 172.21.15.139:0/3068517571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59d00f5580 0x7f59d00f59a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f59c0004660 tx=0x7f59c0009b30).stop 2021-11-20T06:07:40.886 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.893+0000 7f59d674b700 1 -- 172.21.15.139:0/3068517571 shutdown_connections 2021-11-20T06:07:40.886 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.893+0000 7f59d674b700 1 --2- 172.21.15.139:0/3068517571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59d00f6820 0x7f59d00f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:40.886 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.893+0000 7f59d674b700 1 --2- 172.21.15.139:0/3068517571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59d00f5580 0x7f59d00f59a0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:40.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.893+0000 7f59d674b700 1 -- 172.21.15.139:0/3068517571 >> 172.21.15.139:0/3068517571 conn(0x7f59d00f0ce0 msgr2=0x7f59d00f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:40.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.893+0000 7f59d674b700 1 -- 172.21.15.139:0/3068517571 shutdown_connections 2021-11-20T06:07:40.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.893+0000 7f59d674b700 1 -- 172.21.15.139:0/3068517571 wait complete. 2021-11-20T06:07:40.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.893+0000 7f59d674b700 1 Processor -- start 2021-11-20T06:07:40.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.894+0000 7f59d674b700 1 -- start start 2021-11-20T06:07:40.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.894+0000 7f59d674b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59d00f5580 0x7f59d0105db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:40.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.894+0000 7f59d674b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59d00f6820 0x7f59d01062a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:40.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.894+0000 7f59d674b700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d0106850 con 0x7f59d00f6820 2021-11-20T06:07:40.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.894+0000 7f59d674b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d0106990 con 0x7f59d00f5580 2021-11-20T06:07:40.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.894+0000 7f59cf7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59d00f6820 0x7f59d01062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:40.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.894+0000 7f59cf7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59d00f6820 0x7f59d01062a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58890/0 (socket says 172.21.15.139:58890) 2021-11-20T06:07:40.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.894+0000 7f59cf7fe700 1 -- 172.21.15.139:0/4024716061 learned_addr learned my addr 172.21.15.139:0/4024716061 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:40.890 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.894+0000 7f59cffff700 1 --2- 172.21.15.139:0/4024716061 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59d00f5580 0x7f59d0105db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:40.890 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.894+0000 7f59cf7fe700 1 -- 172.21.15.139:0/4024716061 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59d00f5580 msgr2=0x7f59d0105db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:40.890 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.894+0000 7f59cf7fe700 1 --2- 172.21.15.139:0/4024716061 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59d00f5580 0x7f59d0105db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:40.890 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.894+0000 7f59cf7fe700 1 -- 172.21.15.139:0/4024716061 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c0005040 con 0x7f59d00f6820 2021-11-20T06:07:40.891 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.894+0000 7f59cffff700 1 --2- 172.21.15.139:0/4024716061 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59d00f5580 0x7f59d0105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:07:40.891 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.894+0000 7f59cf7fe700 1 --2- 172.21.15.139:0/4024716061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59d00f6820 0x7f59d01062a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f59c40026d0 tx=0x7f59c4009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:40.891 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.894+0000 7f59ccff9700 1 -- 172.21.15.139:0/4024716061 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59c4009320 con 0x7f59d00f6820 2021-11-20T06:07:40.891 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.895+0000 7f59ccff9700 1 -- 172.21.15.139:0/4024716061 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f59c400f040 con 0x7f59d00f6820 2021-11-20T06:07:40.892 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.895+0000 7f59d674b700 1 -- 172.21.15.139:0/4024716061 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d010b430 con 0x7f59d00f6820 2021-11-20T06:07:40.892 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.895+0000 7f59ccff9700 1 -- 172.21.15.139:0/4024716061 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59c40147e0 con 0x7f59d00f6820 2021-11-20T06:07:40.892 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.895+0000 7f59d674b700 1 -- 172.21.15.139:0/4024716061 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d010b9f0 con 0x7f59d00f6820 2021-11-20T06:07:40.892 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.895+0000 7f59d674b700 1 -- 172.21.15.139:0/4024716061 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d0027a00 con 0x7f59d00f6820 2021-11-20T06:07:40.893 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.896+0000 7f59ccff9700 1 -- 172.21.15.139:0/4024716061 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f59c4009480 con 0x7f59d00f6820 2021-11-20T06:07:40.893 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.896+0000 7f59ccff9700 1 --2- 172.21.15.139:0/4024716061 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f59b805c2a0 0x7f59b805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:40.893 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.896+0000 7f59ccff9700 1 -- 172.21.15.139:0/4024716061 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f59c40769f0 con 0x7f59d00f6820 2021-11-20T06:07:40.893 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.896+0000 7f59cffff700 1 --2- 172.21.15.139:0/4024716061 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f59b805c2a0 0x7f59b805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:40.894 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.897+0000 7f59cffff700 1 --2- 172.21.15.139:0/4024716061 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f59b805c2a0 0x7f59b805e760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f59c0009d60 tx=0x7f59c0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:40.894 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:40.900+0000 7f59ccff9700 1 -- 172.21.15.139:0/4024716061 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59c4078e60 con 0x7f59d00f6820 2021-11-20T06:07:41.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:41.082+0000 7f59d674b700 1 -- 172.21.15.139:0/4024716061 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59d00fdd10 con 0x7f59d00f6820 2021-11-20T06:07:41.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:41.083+0000 7f59ccff9700 1 -- 172.21.15.139:0/4024716061 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f59c4049280 con 0x7f59d00f6820 2021-11-20T06:07:41.068 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:41.068 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:30.359623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24489":{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:41.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:41.084+0000 7f59d674b700 1 -- 172.21.15.139:0/4024716061 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f59b805c2a0 msgr2=0x7f59b805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:41.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:41.084+0000 7f59d674b700 1 --2- 172.21.15.139:0/4024716061 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f59b805c2a0 0x7f59b805e760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f59c0009d60 tx=0x7f59c0009b30).stop 2021-11-20T06:07:41.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:41.084+0000 7f59d674b700 1 -- 172.21.15.139:0/4024716061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59d00f6820 msgr2=0x7f59d01062a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:41.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:41.084+0000 7f59d674b700 1 --2- 172.21.15.139:0/4024716061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59d00f6820 0x7f59d01062a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f59c40026d0 tx=0x7f59c4009de0).stop 2021-11-20T06:07:41.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:41.084+0000 7f59d674b700 1 -- 172.21.15.139:0/4024716061 shutdown_connections 2021-11-20T06:07:41.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:41.084+0000 7f59d674b700 1 --2- 172.21.15.139:0/4024716061 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f59b805c2a0 0x7f59b805e760 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:41.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:41.084+0000 7f59d674b700 1 --2- 172.21.15.139:0/4024716061 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59d00f5580 0x7f59d0105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:41.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:41.084+0000 7f59d674b700 1 --2- 172.21.15.139:0/4024716061 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59d00f6820 0x7f59d01062a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:41.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:41.084+0000 7f59d674b700 1 -- 172.21.15.139:0/4024716061 >> 172.21.15.139:0/4024716061 conn(0x7f59d00f0ce0 msgr2=0x7f59d00fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:41.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:41.084+0000 7f59d674b700 1 -- 172.21.15.139:0/4024716061 shutdown_connections 2021-11-20T06:07:41.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:41.084+0000 7f59d674b700 1 -- 172.21.15.139:0/4024716061 wait complete. 2021-11-20T06:07:41.072 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 77 2021-11-20T06:07:41.081 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:30.359623+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24489': {'gid': 24489, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.178:6827/2887891857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2887891857}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2887891857}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:41.081 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24489:up:rejoin 2021-11-20T06:07:41.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:41 smithi178 conmon[36600]: cluster 2021-11-20T06:07:40.274514+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:07:41.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:41 smithi178 conmon[36600]: mgr.14162) 316 : cluster [DBG] pgmap v291: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:07:41.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:41 smithi178 conmon[36600]: audit 2021-11-20T06:07:41.083489+0000 2021-11-20T06:07:41.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:41 smithi178 conmon[36600]: mon.smithi139 (mon.0) 828 : audit [DBG] from='client.? 172.21.15.139:0/4024716061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:41.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:41 smithi139 conmon[33501]: cluster 2021-11-20T06:07:40.274514+0000 mgr.smithi139.dplyxn (mgr.14162) 316 : cluster 2021-11-20T06:07:41.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:41 smithi139 conmon[33501]: [DBG] pgmap v291: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:07:41.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:41 smithi139 conmon[33501]: audit 2021-11-20T06:07:41.083489+0000 mon.smithi139 (mon.0) 828 : audit [DBG] from='client.? 172.21.15.139:0/4024716061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:42.082 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:42.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.225+0000 7fd07e835700 1 -- 172.21.15.139:0/3526882444 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0780f7950 msgr2=0x7fd0780f7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:42.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.225+0000 7fd07e835700 1 --2- 172.21.15.139:0/3526882444 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0780f7950 0x7fd0780f7db0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fd06c004660 tx=0x7fd06c009b30).stop 2021-11-20T06:07:42.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.225+0000 7fd07e835700 1 -- 172.21.15.139:0/3526882444 shutdown_connections 2021-11-20T06:07:42.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.225+0000 7fd07e835700 1 --2- 172.21.15.139:0/3526882444 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0780f7950 0x7fd0780f7db0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:42.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.225+0000 7fd07e835700 1 --2- 172.21.15.139:0/3526882444 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0780f66b0 0x7fd0780f6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:42.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.225+0000 7fd07e835700 1 -- 172.21.15.139:0/3526882444 >> 172.21.15.139:0/3526882444 conn(0x7fd0780f1dd0 msgr2=0x7fd0780f4230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:42.214 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.225+0000 7fd07e835700 1 -- 172.21.15.139:0/3526882444 shutdown_connections 2021-11-20T06:07:42.214 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.225+0000 7fd07e835700 1 -- 172.21.15.139:0/3526882444 wait complete. 2021-11-20T06:07:42.214 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.226+0000 7fd07e835700 1 Processor -- start 2021-11-20T06:07:42.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.226+0000 7fd07e835700 1 -- start start 2021-11-20T06:07:42.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.226+0000 7fd07e835700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0780f66b0 0x7fd078108ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:42.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.226+0000 7fd07e835700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0780f7950 0x7fd0781094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:42.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.226+0000 7fd07e835700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd078109a90 con 0x7fd0780f7950 2021-11-20T06:07:42.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.226+0000 7fd07e835700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd078109bd0 con 0x7fd0780f66b0 2021-11-20T06:07:42.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.226+0000 7fd0777fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0780f7950 0x7fd0781094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:42.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.226+0000 7fd0777fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0780f7950 0x7fd0781094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58900/0 (socket says 172.21.15.139:58900) 2021-11-20T06:07:42.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.226+0000 7fd0777fe700 1 -- 172.21.15.139:0/3049866903 learned_addr learned my addr 172.21.15.139:0/3049866903 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:42.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.226+0000 7fd077fff700 1 --2- 172.21.15.139:0/3049866903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0780f66b0 0x7fd078108ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:42.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.226+0000 7fd0777fe700 1 -- 172.21.15.139:0/3049866903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0780f66b0 msgr2=0x7fd078108ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:42.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.226+0000 7fd0777fe700 1 --2- 172.21.15.139:0/3049866903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0780f66b0 0x7fd078108ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:42.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.226+0000 7fd0777fe700 1 -- 172.21.15.139:0/3049866903 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06c005040 con 0x7fd0780f7950 2021-11-20T06:07:42.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.227+0000 7fd0777fe700 1 --2- 172.21.15.139:0/3049866903 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0780f7950 0x7fd0781094e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fd06c000c00 tx=0x7fd06c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:42.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.227+0000 7fd074ff9700 1 -- 172.21.15.139:0/3049866903 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd06c019070 con 0x7fd0780f7950 2021-11-20T06:07:42.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.227+0000 7fd074ff9700 1 -- 172.21.15.139:0/3049866903 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd06c007e70 con 0x7fd0780f7950 2021-11-20T06:07:42.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.227+0000 7fd07e835700 1 -- 172.21.15.139:0/3049866903 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07810e610 con 0x7fd0780f7950 2021-11-20T06:07:42.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.227+0000 7fd074ff9700 1 -- 172.21.15.139:0/3049866903 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd06c014c90 con 0x7fd0780f7950 2021-11-20T06:07:42.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.227+0000 7fd07e835700 1 -- 172.21.15.139:0/3049866903 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07810ebd0 con 0x7fd0780f7950 2021-11-20T06:07:42.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.228+0000 7fd07e835700 1 -- 172.21.15.139:0/3049866903 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd078027a00 con 0x7fd0780f7950 2021-11-20T06:07:42.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.228+0000 7fd074ff9700 1 -- 172.21.15.139:0/3049866903 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd06c025070 con 0x7fd0780f7950 2021-11-20T06:07:42.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.229+0000 7fd074ff9700 1 --2- 172.21.15.139:0/3049866903 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd06005c2a0 0x7fd06005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:42.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.229+0000 7fd074ff9700 1 -- 172.21.15.139:0/3049866903 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fd06c0773d0 con 0x7fd0780f7950 2021-11-20T06:07:42.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.229+0000 7fd077fff700 1 --2- 172.21.15.139:0/3049866903 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd06005c2a0 0x7fd06005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:42.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.229+0000 7fd077fff700 1 --2- 172.21.15.139:0/3049866903 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd06005c2a0 0x7fd06005e760 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fd068009fd0 tx=0x7fd06800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:42.223 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.233+0000 7fd074ff9700 1 -- 172.21.15.139:0/3049866903 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd06c049be0 con 0x7fd0780f7950 2021-11-20T06:07:42.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.417+0000 7fd07e835700 1 -- 172.21.15.139:0/3049866903 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd078101060 con 0x7fd0780f7950 2021-11-20T06:07:42.402 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.418+0000 7fd074ff9700 1 -- 172.21.15.139:0/3049866903 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2187 (secure 0 0 0) 0x7fd06c04d200 con 0x7fd0780f7950 2021-11-20T06:07:42.403 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:42.403 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:30.359623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24489":{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:42.404 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.419+0000 7fd07e835700 1 -- 172.21.15.139:0/3049866903 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd06005c2a0 msgr2=0x7fd06005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:42.404 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.419+0000 7fd07e835700 1 --2- 172.21.15.139:0/3049866903 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd06005c2a0 0x7fd06005e760 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fd068009fd0 tx=0x7fd06800b040).stop 2021-11-20T06:07:42.405 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.419+0000 7fd07e835700 1 -- 172.21.15.139:0/3049866903 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0780f7950 msgr2=0x7fd0781094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:42.405 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.419+0000 7fd07e835700 1 --2- 172.21.15.139:0/3049866903 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0780f7950 0x7fd0781094e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fd06c000c00 tx=0x7fd06c00b040).stop 2021-11-20T06:07:42.405 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.419+0000 7fd07e835700 1 -- 172.21.15.139:0/3049866903 shutdown_connections 2021-11-20T06:07:42.405 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.419+0000 7fd07e835700 1 --2- 172.21.15.139:0/3049866903 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd06005c2a0 0x7fd06005e760 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:42.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.419+0000 7fd07e835700 1 --2- 172.21.15.139:0/3049866903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0780f66b0 0x7fd078108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:42.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.419+0000 7fd07e835700 1 --2- 172.21.15.139:0/3049866903 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0780f7950 0x7fd0781094e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:42.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.419+0000 7fd07e835700 1 -- 172.21.15.139:0/3049866903 >> 172.21.15.139:0/3049866903 conn(0x7fd0780f1dd0 msgr2=0x7fd0780fb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:42.407 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.420+0000 7fd07e835700 1 -- 172.21.15.139:0/3049866903 shutdown_connections 2021-11-20T06:07:42.407 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:42.420+0000 7fd07e835700 1 -- 172.21.15.139:0/3049866903 wait complete. 2021-11-20T06:07:42.407 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 77 2021-11-20T06:07:42.416 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:30.359623+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24489': {'gid': 24489, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.178:6827/2887891857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2887891857}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2887891857}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:42.416 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24489:up:rejoin 2021-11-20T06:07:43.418 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:43.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:43 smithi139 conmon[33501]: cluster 2021-11-20T06:07:42.274914+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:07:43.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:43 smithi139 conmon[33501]: ) 317 : cluster [DBG] pgmap v292: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T06:07:43.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:43 smithi139 conmon[33501]: audit 2021-11-20T06:07:42.418666+0000 mon.smithi139 (mon.0) 829 : audit [DBG] from='client.? 172.21.15.139:0/3049866903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:43.535 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.549+0000 7fa0913fe700 1 -- 172.21.15.139:0/857454727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa08c0f7870 msgr2=0x7fa08c0f7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:43.536 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.549+0000 7fa0913fe700 1 --2- 172.21.15.139:0/857454727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa08c0f7870 0x7fa08c0f7cd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fa080004660 tx=0x7fa080009b30).stop 2021-11-20T06:07:43.536 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.549+0000 7fa0913fe700 1 -- 172.21.15.139:0/857454727 shutdown_connections 2021-11-20T06:07:43.536 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.549+0000 7fa0913fe700 1 --2- 172.21.15.139:0/857454727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa08c0f7870 0x7fa08c0f7cd0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:43.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.549+0000 7fa0913fe700 1 --2- 172.21.15.139:0/857454727 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa08c0f65d0 0x7fa08c0f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:43.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.549+0000 7fa0913fe700 1 -- 172.21.15.139:0/857454727 >> 172.21.15.139:0/857454727 conn(0x7fa08c0f1d10 msgr2=0x7fa08c0f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:43.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.549+0000 7fa0913fe700 1 -- 172.21.15.139:0/857454727 shutdown_connections 2021-11-20T06:07:43.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.549+0000 7fa0913fe700 1 -- 172.21.15.139:0/857454727 wait complete. 2021-11-20T06:07:43.537 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.550+0000 7fa0913fe700 1 Processor -- start 2021-11-20T06:07:43.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.550+0000 7fa0913fe700 1 -- start start 2021-11-20T06:07:43.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.550+0000 7fa0913fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa08c0f65d0 0x7fa08c106d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:43.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.550+0000 7fa0913fe700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa08c0f7870 0x7fa08c107280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:43.538 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.550+0000 7fa0913fe700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c107880 con 0x7fa08c0f65d0 2021-11-20T06:07:43.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.550+0000 7fa0913fe700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c108090 con 0x7fa08c0f7870 2021-11-20T06:07:43.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.550+0000 7fa08bfff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa08c0f65d0 0x7fa08c106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:43.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.550+0000 7fa08bfff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa08c0f65d0 0x7fa08c106d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58906/0 (socket says 172.21.15.139:58906) 2021-11-20T06:07:43.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.550+0000 7fa08bfff700 1 -- 172.21.15.139:0/3091560562 learned_addr learned my addr 172.21.15.139:0/3091560562 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:43.539 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.550+0000 7fa08b7fe700 1 --2- 172.21.15.139:0/3091560562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa08c0f7870 0x7fa08c107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:43.540 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.550+0000 7fa08bfff700 1 -- 172.21.15.139:0/3091560562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa08c0f7870 msgr2=0x7fa08c107280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:43.540 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.550+0000 7fa08bfff700 1 --2- 172.21.15.139:0/3091560562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa08c0f7870 0x7fa08c107280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:43.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.550+0000 7fa08bfff700 1 -- 172.21.15.139:0/3091560562 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa080005040 con 0x7fa08c0f65d0 2021-11-20T06:07:43.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.550+0000 7fa08bfff700 1 --2- 172.21.15.139:0/3091560562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa08c0f65d0 0x7fa08c106d90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fa07c00c790 tx=0x7fa07c007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:43.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.551+0000 7fa088ff9700 1 -- 172.21.15.139:0/3091560562 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa07c00e680 con 0x7fa08c0f65d0 2021-11-20T06:07:43.541 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.551+0000 7fa0913fe700 1 -- 172.21.15.139:0/3091560562 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa08c108310 con 0x7fa08c0f65d0 2021-11-20T06:07:43.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.551+0000 7fa088ff9700 1 -- 172.21.15.139:0/3091560562 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa07c00ec90 con 0x7fa08c0f65d0 2021-11-20T06:07:43.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.551+0000 7fa088ff9700 1 -- 172.21.15.139:0/3091560562 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa07c012cb0 con 0x7fa08c0f65d0 2021-11-20T06:07:43.542 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.551+0000 7fa0913fe700 1 -- 172.21.15.139:0/3091560562 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa08c1088d0 con 0x7fa08c0f65d0 2021-11-20T06:07:43.543 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.552+0000 7fa0913fe700 1 -- 172.21.15.139:0/3091560562 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa08c0f59c0 con 0x7fa08c0f65d0 2021-11-20T06:07:43.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.552+0000 7fa088ff9700 1 -- 172.21.15.139:0/3091560562 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa07c010070 con 0x7fa08c0f65d0 2021-11-20T06:07:43.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.552+0000 7fa088ff9700 1 --2- 172.21.15.139:0/3091560562 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa07405c300 0x7fa07405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:43.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.552+0000 7fa088ff9700 1 -- 172.21.15.139:0/3091560562 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa07c076480 con 0x7fa08c0f65d0 2021-11-20T06:07:43.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.552+0000 7fa08b7fe700 1 --2- 172.21.15.139:0/3091560562 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa07405c300 0x7fa07405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:43.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.553+0000 7fa08b7fe700 1 --2- 172.21.15.139:0/3091560562 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa07405c300 0x7fa07405e7c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fa080013580 tx=0x7fa080009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:43.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.556+0000 7fa088ff9700 1 -- 172.21.15.139:0/3091560562 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa07c048c90 con 0x7fa08c0f65d0 2021-11-20T06:07:43.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:43 smithi178 conmon[36600]: cluster 2021-11-20T06:07:42.274914+0000 mgr.smithi139.dplyxn (mgr.14162) 317 : cluster [DBG] pgmap v292: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 426 B/s rd, 0 op/s 2021-11-20T06:07:43.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:43 smithi178 conmon[36600]: audit 2021-11-20T06:07:42.418666+0000 mon.smithi139 (mon.0) 829 : audit [DBG] from='client.? 172.21.15.139:0/3049866903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:43.725 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.741+0000 7fa0913fe700 1 -- 172.21.15.139:0/3091560562 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa08c045ad0 con 0x7fa08c0f65d0 2021-11-20T06:07:43.726 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.741+0000 7fa088ff9700 1 -- 172.21.15.139:0/3091560562 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2187 (secure 0 0 0) 0x7fa07c04c2b0 con 0x7fa08c0f65d0 2021-11-20T06:07:43.727 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:43.727 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:30.359623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24489":{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:43.728 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.743+0000 7fa0913fe700 1 -- 172.21.15.139:0/3091560562 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa07405c300 msgr2=0x7fa07405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:43.728 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.743+0000 7fa0913fe700 1 --2- 172.21.15.139:0/3091560562 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa07405c300 0x7fa07405e7c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fa080013580 tx=0x7fa080009b30).stop 2021-11-20T06:07:43.729 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.743+0000 7fa0913fe700 1 -- 172.21.15.139:0/3091560562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa08c0f65d0 msgr2=0x7fa08c106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:43.729 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.743+0000 7fa0913fe700 1 --2- 172.21.15.139:0/3091560562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa08c0f65d0 0x7fa08c106d90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fa07c00c790 tx=0x7fa07c007d90).stop 2021-11-20T06:07:43.729 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.743+0000 7fa0913fe700 1 -- 172.21.15.139:0/3091560562 shutdown_connections 2021-11-20T06:07:43.729 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.743+0000 7fa0913fe700 1 --2- 172.21.15.139:0/3091560562 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa07405c300 0x7fa07405e7c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:43.730 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.743+0000 7fa0913fe700 1 --2- 172.21.15.139:0/3091560562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa08c0f65d0 0x7fa08c106d90 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:43.730 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.743+0000 7fa0913fe700 1 --2- 172.21.15.139:0/3091560562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa08c0f7870 0x7fa08c107280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:43.730 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.743+0000 7fa0913fe700 1 -- 172.21.15.139:0/3091560562 >> 172.21.15.139:0/3091560562 conn(0x7fa08c0f1d10 msgr2=0x7fa08c0fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:43.730 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.743+0000 7fa0913fe700 1 -- 172.21.15.139:0/3091560562 shutdown_connections 2021-11-20T06:07:43.730 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:43.743+0000 7fa0913fe700 1 -- 172.21.15.139:0/3091560562 wait complete. 2021-11-20T06:07:43.731 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 77 2021-11-20T06:07:43.739 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:30.359623+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24489': {'gid': 24489, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.178:6827/2887891857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2887891857}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2887891857}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:43.740 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24489:up:rejoin 2021-11-20T06:07:44.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:44 smithi139 conmon[33501]: audit 2021-11-20T06:07:43.742151+0000 mon.smithi139 (mon.0) 830 : audit 2021-11-20T06:07:44.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:44 smithi139 conmon[33501]: [DBG] from='client.? 172.21.15.139:0/3091560562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:44.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:44 smithi178 conmon[36600]: audit 2021-11-20T06:07:43.742151+0000 mon.smithi139 (mon. 2021-11-20T06:07:44.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:44 smithi178 conmon[36600]: 0) 830 : audit [DBG] from='client.? 172.21.15.139:0/3091560562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:44.741 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:44.870 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.884+0000 7fa8219aa700 1 -- 172.21.15.139:0/563486817 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa81c0f5580 msgr2=0x7fa81c0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:44.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.884+0000 7fa8219aa700 1 --2- 172.21.15.139:0/563486817 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa81c0f5580 0x7fa81c0f59a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fa80c004660 tx=0x7fa80c009b30).stop 2021-11-20T06:07:44.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.884+0000 7fa8219aa700 1 -- 172.21.15.139:0/563486817 shutdown_connections 2021-11-20T06:07:44.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.884+0000 7fa8219aa700 1 --2- 172.21.15.139:0/563486817 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa81c0f6820 0x7fa81c0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:44.871 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.884+0000 7fa8219aa700 1 --2- 172.21.15.139:0/563486817 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa81c0f5580 0x7fa81c0f59a0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:44.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.884+0000 7fa8219aa700 1 -- 172.21.15.139:0/563486817 >> 172.21.15.139:0/563486817 conn(0x7fa81c0f0ce0 msgr2=0x7fa81c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:44.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.884+0000 7fa8219aa700 1 -- 172.21.15.139:0/563486817 shutdown_connections 2021-11-20T06:07:44.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.884+0000 7fa8219aa700 1 -- 172.21.15.139:0/563486817 wait complete. 2021-11-20T06:07:44.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.885+0000 7fa8219aa700 1 Processor -- start 2021-11-20T06:07:44.872 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.885+0000 7fa8219aa700 1 -- start start 2021-11-20T06:07:44.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.885+0000 7fa8219aa700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa81c0f5580 0x7fa81c060550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:44.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.885+0000 7fa8219aa700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa81c0f6820 0x7fa81c10e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:44.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.885+0000 7fa8219aa700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c107ac0 con 0x7fa81c0f6820 2021-11-20T06:07:44.873 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.885+0000 7fa8219aa700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c10e5d0 con 0x7fa81c0f5580 2021-11-20T06:07:44.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.885+0000 7fa81a7fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa81c0f6820 0x7fa81c10e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:44.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.885+0000 7fa81a7fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa81c0f6820 0x7fa81c10e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58918/0 (socket says 172.21.15.139:58918) 2021-11-20T06:07:44.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.885+0000 7fa81a7fc700 1 -- 172.21.15.139:0/764258315 learned_addr learned my addr 172.21.15.139:0/764258315 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:44.874 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.885+0000 7fa81a7fc700 1 -- 172.21.15.139:0/764258315 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa81c0f5580 msgr2=0x7fa81c060550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:44.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.885+0000 7fa81a7fc700 1 --2- 172.21.15.139:0/764258315 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa81c0f5580 0x7fa81c060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:07:44.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.885+0000 7fa81a7fc700 1 -- 172.21.15.139:0/764258315 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80c005040 con 0x7fa81c0f6820 2021-11-20T06:07:44.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.885+0000 7fa81a7fc700 1 --2- 172.21.15.139:0/764258315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa81c0f6820 0x7fa81c10e0e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fa81000c530 tx=0x7fa810007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:44.875 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.886+0000 7fa8037fe700 1 -- 172.21.15.139:0/764258315 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa81000f650 con 0x7fa81c0f6820 2021-11-20T06:07:44.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.886+0000 7fa8037fe700 1 -- 172.21.15.139:0/764258315 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa81000fc60 con 0x7fa81c0f6820 2021-11-20T06:07:44.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.886+0000 7fa8219aa700 1 -- 172.21.15.139:0/764258315 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81c10e770 con 0x7fa81c0f6820 2021-11-20T06:07:44.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.886+0000 7fa8037fe700 1 -- 172.21.15.139:0/764258315 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa810012c80 con 0x7fa81c0f6820 2021-11-20T06:07:44.876 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.886+0000 7fa8219aa700 1 -- 172.21.15.139:0/764258315 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81c10ed00 con 0x7fa81c0f6820 2021-11-20T06:07:44.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.887+0000 7fa8219aa700 1 -- 172.21.15.139:0/764258315 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81c046330 con 0x7fa81c0f6820 2021-11-20T06:07:44.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.887+0000 7fa8037fe700 1 -- 172.21.15.139:0/764258315 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa810010070 con 0x7fa81c0f6820 2021-11-20T06:07:44.878 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.887+0000 7fa8037fe700 1 --2- 172.21.15.139:0/764258315 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa80405c2a0 0x7fa80405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:44.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.887+0000 7fa8037fe700 1 -- 172.21.15.139:0/764258315 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa810075ff0 con 0x7fa81c0f6820 2021-11-20T06:07:44.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.887+0000 7fa81affd700 1 --2- 172.21.15.139:0/764258315 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa80405c2a0 0x7fa80405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:44.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.888+0000 7fa81affd700 1 --2- 172.21.15.139:0/764258315 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa80405c2a0 0x7fa80405e760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fa80c009f50 tx=0x7fa80c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:44.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:44.891+0000 7fa8037fe700 1 -- 172.21.15.139:0/764258315 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8100452b0 con 0x7fa81c0f6820 2021-11-20T06:07:45.061 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:45.077+0000 7fa8219aa700 1 -- 172.21.15.139:0/764258315 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa81c027a00 con 0x7fa81c0f6820 2021-11-20T06:07:45.062 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:45.077+0000 7fa8037fe700 1 -- 172.21.15.139:0/764258315 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2187 (secure 0 0 0) 0x7fa81004c2b0 con 0x7fa81c0f6820 2021-11-20T06:07:45.062 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:45.063 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:30.359623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24489},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24489":{"gid":24489,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":74,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.178:6827/2887891857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2887891857},{"type":"v1","addr":"172.21.15.178:6827","nonce":2887891857}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:45.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:45.079+0000 7fa8219aa700 1 -- 172.21.15.139:0/764258315 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa80405c2a0 msgr2=0x7fa80405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:45.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:45.079+0000 7fa8219aa700 1 --2- 172.21.15.139:0/764258315 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa80405c2a0 0x7fa80405e760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fa80c009f50 tx=0x7fa80c009b30).stop 2021-11-20T06:07:45.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:45.079+0000 7fa8219aa700 1 -- 172.21.15.139:0/764258315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa81c0f6820 msgr2=0x7fa81c10e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:45.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:45.079+0000 7fa8219aa700 1 --2- 172.21.15.139:0/764258315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa81c0f6820 0x7fa81c10e0e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fa81000c530 tx=0x7fa810007e20).stop 2021-11-20T06:07:45.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:45.079+0000 7fa8219aa700 1 -- 172.21.15.139:0/764258315 shutdown_connections 2021-11-20T06:07:45.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:45.079+0000 7fa8219aa700 1 --2- 172.21.15.139:0/764258315 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa80405c2a0 0x7fa80405e760 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:45.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:45.079+0000 7fa8219aa700 1 --2- 172.21.15.139:0/764258315 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa81c0f5580 0x7fa81c060550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:07:45.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:45.079+0000 7fa8219aa700 1 --2- 172.21.15.139:0/764258315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa81c0f6820 0x7fa81c10e0e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:45.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:45.079+0000 7fa8219aa700 1 -- 172.21.15.139:0/764258315 >> 172.21.15.139:0/764258315 conn(0x7fa81c0f0ce0 msgr2=0x7fa81c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:45.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:45.079+0000 7fa8219aa700 1 -- 172.21.15.139:0/764258315 shutdown_connections 2021-11-20T06:07:45.066 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:45.079+0000 7fa8219aa700 1 -- 172.21.15.139:0/764258315 wait complete. 2021-11-20T06:07:45.067 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 77 2021-11-20T06:07:45.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:30.359623+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24489}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24489': {'gid': 24489, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 74, 'state': 'up:rejoin', 'state_seq': 6, 'addr': '172.21.15.178:6827/2887891857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2887891857}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2887891857}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:45.076 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24489:up:rejoin 2021-11-20T06:07:45.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:45 smithi139 conmon[33501]: cluster 2021-11-20T06:07:44.275431+0000 mgr.smithi139.dplyxn (mgr.14162) 318 : cluster [DBG] pgmap v293: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:07:45.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:45 smithi139 conmon[33501]: 2021-11-20T06:07:45.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:45 smithi139 conmon[33501]: audit 2021-11-20T06:07:45.078203+0000 mon.smithi139 (mon.0) 831 : audit [DBG] from='client.? 172.21.15.139:0/764258315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:45.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:45 smithi178 conmon[36600]: cluster 2021-11-20T06:07:44.275431+0000 mgr.smithi139.dplyxn (mgr 2021-11-20T06:07:45.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:45 smithi178 conmon[36600]: .14162) 318 : cluster [DBG] pgmap v293: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:07:45.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:45 smithi178 conmon[36600]: audit 2021-11-20T06:07:45.078203+0000 mon.smithi139 (mon.0) 831 : audit [DBG] from='client.? 172.21.15.139:0/764258315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:46.077 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:46.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.219+0000 7f0c91577700 1 -- 172.21.15.139:0/4254800750 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c8c0f5460 msgr2=0x7f0c8c0f5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:46.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.219+0000 7f0c91577700 1 --2- 172.21.15.139:0/4254800750 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c8c0f5460 0x7f0c8c0f5880 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f0c7c004660 tx=0x7f0c7c009b30).stop 2021-11-20T06:07:46.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.219+0000 7f0c91577700 1 -- 172.21.15.139:0/4254800750 shutdown_connections 2021-11-20T06:07:46.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.219+0000 7f0c91577700 1 --2- 172.21.15.139:0/4254800750 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c8c0f6700 0x7f0c8c0f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:07:46.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.219+0000 7f0c91577700 1 --2- 172.21.15.139:0/4254800750 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c8c0f5460 0x7f0c8c0f5880 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:46.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.219+0000 7f0c91577700 1 -- 172.21.15.139:0/4254800750 >> 172.21.15.139:0/4254800750 conn(0x7f0c8c0f0ba0 msgr2=0x7f0c8c0f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:46.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.219+0000 7f0c91577700 1 -- 172.21.15.139:0/4254800750 shutdown_connections 2021-11-20T06:07:46.211 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.219+0000 7f0c91577700 1 -- 172.21.15.139:0/4254800750 wait complete. 2021-11-20T06:07:46.211 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.219+0000 7f0c91577700 1 Processor -- start 2021-11-20T06:07:46.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c91577700 1 -- start start 2021-11-20T06:07:46.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c91577700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c8c0f5460 0x7f0c8c109a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:46.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c91577700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c8c0f6700 0x7f0c8c10bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:46.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c91577700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c0602a0 con 0x7f0c8c0f6700 2021-11-20T06:07:46.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c91577700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c0603e0 con 0x7f0c8c0f5460 2021-11-20T06:07:46.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c8b7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c8c0f6700 0x7f0c8c10bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:46.214 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c8b7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c8c0f6700 0x7f0c8c10bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58928/0 (socket says 172.21.15.139:58928) 2021-11-20T06:07:46.214 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c8b7fe700 1 -- 172.21.15.139:0/3343627172 learned_addr learned my addr 172.21.15.139:0/3343627172 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:46.215 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c8bfff700 1 --2- 172.21.15.139:0/3343627172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c8c0f5460 0x7f0c8c109a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:46.215 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c8b7fe700 1 -- 172.21.15.139:0/3343627172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c8c0f5460 msgr2=0x7f0c8c109a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:46.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c8b7fe700 1 --2- 172.21.15.139:0/3343627172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c8c0f5460 0x7f0c8c109a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:46.216 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c8b7fe700 1 -- 172.21.15.139:0/3343627172 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7c005040 con 0x7f0c8c0f6700 2021-11-20T06:07:46.217 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c8b7fe700 1 --2- 172.21.15.139:0/3343627172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c8c0f6700 0x7f0c8c10bf70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f0c8000b410 tx=0x7f0c80006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:46.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c88ff9700 1 -- 172.21.15.139:0/3343627172 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c800125f0 con 0x7f0c8c0f6700 2021-11-20T06:07:46.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c91577700 1 -- 172.21.15.139:0/3343627172 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8c10c5d0 con 0x7f0c8c0f6700 2021-11-20T06:07:46.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c88ff9700 1 -- 172.21.15.139:0/3343627172 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c80012c00 con 0x7f0c8c0f6700 2021-11-20T06:07:46.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c88ff9700 1 -- 172.21.15.139:0/3343627172 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c80002aa0 con 0x7f0c8c0f6700 2021-11-20T06:07:46.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.220+0000 7f0c91577700 1 -- 172.21.15.139:0/3343627172 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8c10cb60 con 0x7f0c8c0f6700 2021-11-20T06:07:46.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.221+0000 7f0c91577700 1 -- 172.21.15.139:0/3343627172 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c8c0f4850 con 0x7f0c8c0f6700 2021-11-20T06:07:46.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.222+0000 7f0c88ff9700 1 -- 172.21.15.139:0/3343627172 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0c8000f030 con 0x7f0c8c0f6700 2021-11-20T06:07:46.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.222+0000 7f0c88ff9700 1 --2- 172.21.15.139:0/3343627172 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0c7405c300 0x7f0c7405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:46.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.222+0000 7f0c88ff9700 1 -- 172.21.15.139:0/3343627172 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f0c800768a0 con 0x7f0c8c0f6700 2021-11-20T06:07:46.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.222+0000 7f0c8bfff700 1 --2- 172.21.15.139:0/3343627172 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0c7405c300 0x7f0c7405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:46.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.223+0000 7f0c8bfff700 1 --2- 172.21.15.139:0/3343627172 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0c7405c300 0x7f0c7405e7c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f0c8c05fe90 tx=0x7f0c7c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:46.223 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.226+0000 7f0c88ff9700 1 -- 172.21.15.139:0/3343627172 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c80049000 con 0x7f0c8c0f6700 2021-11-20T06:07:46.403 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.419+0000 7f0c91577700 1 -- 172.21.15.139:0/3343627172 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c8c045ad0 con 0x7f0c8c0f6700 2021-11-20T06:07:46.404 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.419+0000 7f0c88ff9700 1 -- 172.21.15.139:0/3343627172 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f0c8004c620 con 0x7f0c8c0f6700 2021-11-20T06:07:46.404 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:46.405 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":79,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:46.260845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24513},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24513":{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":79,"state":"up:replay","state_seq":2,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:46.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.421+0000 7f0c91577700 1 -- 172.21.15.139:0/3343627172 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0c7405c300 msgr2=0x7f0c7405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:46.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.421+0000 7f0c91577700 1 --2- 172.21.15.139:0/3343627172 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0c7405c300 0x7f0c7405e7c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f0c8c05fe90 tx=0x7f0c7c009b30).stop 2021-11-20T06:07:46.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.421+0000 7f0c91577700 1 -- 172.21.15.139:0/3343627172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c8c0f6700 msgr2=0x7f0c8c10bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:46.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.421+0000 7f0c91577700 1 --2- 172.21.15.139:0/3343627172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c8c0f6700 0x7f0c8c10bf70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f0c8000b410 tx=0x7f0c80006ef0).stop 2021-11-20T06:07:46.407 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.421+0000 7f0c91577700 1 -- 172.21.15.139:0/3343627172 shutdown_connections 2021-11-20T06:07:46.407 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.421+0000 7f0c91577700 1 --2- 172.21.15.139:0/3343627172 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f0c7405c300 0x7f0c7405e7c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:46.407 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.421+0000 7f0c91577700 1 --2- 172.21.15.139:0/3343627172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c8c0f5460 0x7f0c8c109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:46.407 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.421+0000 7f0c91577700 1 --2- 172.21.15.139:0/3343627172 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c8c0f6700 0x7f0c8c10bf70 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:46.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.421+0000 7f0c91577700 1 -- 172.21.15.139:0/3343627172 >> 172.21.15.139:0/3343627172 conn(0x7f0c8c0f0ba0 msgr2=0x7f0c8c0fa150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:46.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.421+0000 7f0c91577700 1 -- 172.21.15.139:0/3343627172 shutdown_connections 2021-11-20T06:07:46.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:46.421+0000 7f0c91577700 1 -- 172.21.15.139:0/3343627172 wait complete. 2021-11-20T06:07:46.408 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 79 2021-11-20T06:07:46.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:46 smithi139 conmon[33501]: debug 2021-11-20T06:07:46.254+0000 7f2a21654700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:07:46.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:46 smithi139 conmon[33501]: cluster 2021-11-20T 2021-11-20T06:07:46.417 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:46 smithi139 conmon[33501]: 06:07:45.269539+0000 mon.smithi139 (mon.0) 832 : cluster [INF] daemon mds.cephfs.smithi178.pskihy restarted 2021-11-20T06:07:46.417 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:46 smithi139 conmon[33501]: cluster 2021-11-20T06:07:45. 2021-11-20T06:07:46.417 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:46 smithi139 conmon[33501]: 273623+0000 mon.smithi139 (mon.0) 833 : cluster [DBG] 2021-11-20T06:07:46.417 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:46 smithi139 conmon[33501]: osdmap e56: 6 total, 6 up, 6 in 2021-11-20T06:07:46.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:46.260845+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24513}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24513': {'gid': 24513, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 79, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.178:6825/1490972326', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 1490972326}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 1490972326}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:46.419 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24513:up:replay 2021-11-20T06:07:46.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:46 smithi178 conmon[36600]: cluster 2021-11-20T06:07:45.269539+0000 mon.smithi139 (mon.0) 832 : cluster 2021-11-20T06:07:46.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:46 smithi178 conmon[36600]: [INF] daemon mds.cephfs.smithi178.pskihy restarted 2021-11-20T06:07:46.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:46 smithi178 conmon[36600]: cluster 2021-11-20T06:07:45.273623+0000 mon.smithi139 (mon.0) 833 : cluster [DBG] osdmap e56: 6 total, 6 up, 6 in 2021-11-20T06:07:47.420 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:47.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: cluster 2021-11-20T06:07:46.255228+0000 mon.smithi139 (mon.0) 834 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:07:47.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: cluster 2021-11-20T06:07:46.260755+0000 mon.smithi139 (mon.0) 835 : cluster [DBG] mds.? [v2:172.21.15.178:6826/2304740608,v1:172.21.15.178:6827/2304740608] up:boot 2021-11-20T06:07:47.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: cluster 2021-11-20T06:07:46. 2021-11-20T06:07:47.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: 260813+0000 mon.smithi139 (mon.0) 836 : cluster [INF] Standby daemon mds.cephfs.smithi178.blvkbx assigned to filesystem cephfs as rank 0 2021-11-20T06:07:47.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: cluster 2021-11-20 2021-11-20T06:07:47.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: T06:07:46.260899+0000 mon.smithi139 ( 2021-11-20T06:07:47.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: mon.0) 837 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:07:47.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: cluster 2021 2021-11-20T06:07:47.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: -11-20T06:07:46 2021-11-20T06:07:47.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: .261079+0000 mon.smithi139 (mon.0) 838 : cluster 2021-11-20T06:07:47.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T06:07:47.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: audit 2021-11-20T 2021-11-20T06:07:47.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: 06:07:46.261188+0000 mon.smithi139 (mon 2021-11-20T06:07:47.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: .0) 839 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.pskihy"}]: dispatch 2021-11-20T06:07:47.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: cluster 2021 2021-11-20T06:07:47.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: -11-20T06:07:46 2021-11-20T06:07:47.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: .263513+0000 mon.smithi139 (mon.0) 840 : cluster 2021-11-20T06:07:47.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:replay} 1 up:standby 2021-11-20T06:07:47.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: cluster 2021-11-20 2021-11-20T06:07:47.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: T06:07:46.275851+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:07:47.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: mgr.14162) 319 : cluster [DBG] pgmap v295: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:07:47.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: audit 2021 2021-11-20T06:07:47.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: -11-20T06:07:46. 2021-11-20T06:07:47.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: 420191+0000 mon.smithi139 (mon.0) 841 : audit [DBG] 2021-11-20T06:07:47.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:47 smithi139 conmon[33501]: from='client.? 172.21.15.139:0/3343627172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:47.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.556+0000 7f518759e700 1 -- 172.21.15.139:0/2627989787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51880f54d0 msgr2=0x7f51880f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:47.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.556+0000 7f518759e700 1 --2- 172.21.15.139:0/2627989787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51880f54d0 0x7f51880f58f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f5178004660 tx=0x7f5178009b30).stop 2021-11-20T06:07:47.544 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.556+0000 7f518759e700 1 -- 172.21.15.139:0/2627989787 shutdown_connections 2021-11-20T06:07:47.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.556+0000 7f518759e700 1 --2- 172.21.15.139:0/2627989787 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51880f6770 0x7f51880f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:47.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.556+0000 7f518759e700 1 --2- 172.21.15.139:0/2627989787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51880f54d0 0x7f51880f58f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:47.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.556+0000 7f518759e700 1 -- 172.21.15.139:0/2627989787 >> 172.21.15.139:0/2627989787 conn(0x7f51880f0c10 msgr2=0x7f51880f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:47.545 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.556+0000 7f518759e700 1 -- 172.21.15.139:0/2627989787 shutdown_connections 2021-11-20T06:07:47.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.556+0000 7f518759e700 1 -- 172.21.15.139:0/2627989787 wait complete. 2021-11-20T06:07:47.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.557+0000 7f518759e700 1 Processor -- start 2021-11-20T06:07:47.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.557+0000 7f518759e700 1 -- start start 2021-11-20T06:07:47.546 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.557+0000 7f518759e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51880f54d0 0x7f518810be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:47.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.557+0000 7f518759e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51880f6770 0x7f518810e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:47.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.557+0000 7f518759e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51880603d0 con 0x7f51880f54d0 2021-11-20T06:07:47.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.557+0000 7f518759e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5188060510 con 0x7f51880f6770 2021-11-20T06:07:47.547 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.557+0000 7f518659c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51880f54d0 0x7f518810be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:47.548 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.557+0000 7f518659c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51880f54d0 0x7f518810be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58938/0 (socket says 172.21.15.139:58938) 2021-11-20T06:07:47.548 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.557+0000 7f518659c700 1 -- 172.21.15.139:0/376536621 learned_addr learned my addr 172.21.15.139:0/376536621 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:47.548 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.557+0000 7f5185d9b700 1 --2- 172.21.15.139:0/376536621 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51880f6770 0x7f518810e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:47.548 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.557+0000 7f518659c700 1 -- 172.21.15.139:0/376536621 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51880f6770 msgr2=0x7f518810e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:47.548 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.557+0000 7f518659c700 1 --2- 172.21.15.139:0/376536621 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51880f6770 0x7f518810e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:47.549 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.557+0000 7f518659c700 1 -- 172.21.15.139:0/376536621 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5178005040 con 0x7f51880f54d0 2021-11-20T06:07:47.549 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.557+0000 7f518659c700 1 --2- 172.21.15.139:0/376536621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51880f54d0 0x7f518810be70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f5178004e30 tx=0x7f5178009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:47.549 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.557+0000 7f5176ffd700 1 -- 172.21.15.139:0/376536621 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5178018070 con 0x7f51880f54d0 2021-11-20T06:07:47.549 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.558+0000 7f5176ffd700 1 -- 172.21.15.139:0/376536621 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5178013410 con 0x7f51880f54d0 2021-11-20T06:07:47.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.558+0000 7f5176ffd700 1 -- 172.21.15.139:0/376536621 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f517801c3c0 con 0x7f51880f54d0 2021-11-20T06:07:47.550 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.558+0000 7f518759e700 1 -- 172.21.15.139:0/376536621 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518810e860 con 0x7f51880f54d0 2021-11-20T06:07:47.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.558+0000 7f518759e700 1 -- 172.21.15.139:0/376536621 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f518810ec30 con 0x7f51880f54d0 2021-11-20T06:07:47.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.558+0000 7f518759e700 1 -- 172.21.15.139:0/376536621 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5188045ad0 con 0x7f51880f54d0 2021-11-20T06:07:47.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.559+0000 7f5176ffd700 1 -- 172.21.15.139:0/376536621 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f517801d420 con 0x7f51880f54d0 2021-11-20T06:07:47.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.559+0000 7f5176ffd700 1 --2- 172.21.15.139:0/376536621 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f517005c300 0x7f517005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:47.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.559+0000 7f5176ffd700 1 -- 172.21.15.139:0/376536621 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f5178078010 con 0x7f51880f54d0 2021-11-20T06:07:47.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.559+0000 7f5185d9b700 1 --2- 172.21.15.139:0/376536621 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f517005c300 0x7f517005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:47.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.559+0000 7f5185d9b700 1 --2- 172.21.15.139:0/376536621 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f517005c300 0x7f517005e7c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f517c0096c0 tx=0x7f517c009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:47.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.563+0000 7f5176ffd700 1 -- 172.21.15.139:0/376536621 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f517804a7d0 con 0x7f51880f54d0 2021-11-20T06:07:47.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: cluster 2021-11-20T06:07:46.255228+0000 mon.smithi139 (mon.0) 834 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:07:47.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: cluster 2021-11-20T06:07:46.260755+0000 mon.smithi139 (mon.0) 835 : cluster [DBG] mds.? [v2:172.21.15.178:6826/2304740608,v1:172.21.15.178:6827/2304740608] up:boot 2021-11-20T06:07:47.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: cluster 2021-11-20T06:07:46.260813+0000 mon.smithi139 (mon.0) 836 : cluster [INF] Standby daemon mds.cephfs.smithi178.blvkbx assigned to filesystem cephfs as rank 0 2021-11-20T06:07:47.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: cluster 2021-11-20T06:07:46.260899+0000 mon.smithi139 (mon.0) 837 2021-11-20T06:07:47.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:07:47.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: cluster 2021-11-20T06:07:47.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: 2021-11-20T06:07:46.261079+0000 mon.smithi139 (mon. 2021-11-20T06:07:47.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: 0) 838 : cluster [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T06:07:47.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: audit 2021-11-20T06:07:46 2021-11-20T06:07:47.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: .261188+0000 mon.smithi139 (mon.0) 839 : audit [DBG] 2021-11-20T06:07:47.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.pskihy"}]: dispatch 2021-11-20T06:07:47.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: cluster 2021-11-20T06:07:46.263513+0000 2021-11-20T06:07:47.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: mon.smithi139 (mon.0) 840 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:replay} 1 up:standby 2021-11-20T06:07:47.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: cluster 2021 2021-11-20T06:07:47.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: -11-20T06:07:46.275851+0000 2021-11-20T06:07:47.698 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: mgr.smithi139.dplyxn (mgr.14162) 319 : cluster [DBG] pgmap v295: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:07:47.698 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: 2021-11-20T06:07:47.699 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: audit 2021-11-20T06:07:47.699 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: 2021-11-20T06:07:46.420191+0000 mon.smithi139 2021-11-20T06:07:47.699 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:47 smithi178 conmon[36600]: (mon.0) 841 : audit [DBG] from='client.? 172.21.15.139:0/3343627172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:47.742 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.758+0000 7f518759e700 1 -- 172.21.15.139:0/376536621 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5188027a00 con 0x7f51880f54d0 2021-11-20T06:07:47.743 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.758+0000 7f5176ffd700 1 -- 172.21.15.139:0/376536621 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2191 (secure 0 0 0) 0x7f517804ddf0 con 0x7f51880f54d0 2021-11-20T06:07:47.743 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:47.744 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":80,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:47.262249+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24513},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24513":{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":79,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:47.745 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.760+0000 7f518759e700 1 -- 172.21.15.139:0/376536621 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f517005c300 msgr2=0x7f517005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:47.745 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.760+0000 7f518759e700 1 --2- 172.21.15.139:0/376536621 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f517005c300 0x7f517005e7c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f517c0096c0 tx=0x7f517c009be0).stop 2021-11-20T06:07:47.745 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.760+0000 7f518759e700 1 -- 172.21.15.139:0/376536621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51880f54d0 msgr2=0x7f518810be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:47.745 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.760+0000 7f518759e700 1 --2- 172.21.15.139:0/376536621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51880f54d0 0x7f518810be70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f5178004e30 tx=0x7f5178009b30).stop 2021-11-20T06:07:47.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.760+0000 7f518759e700 1 -- 172.21.15.139:0/376536621 shutdown_connections 2021-11-20T06:07:47.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.760+0000 7f518759e700 1 --2- 172.21.15.139:0/376536621 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f517005c300 0x7f517005e7c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:47.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.760+0000 7f518759e700 1 --2- 172.21.15.139:0/376536621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f51880f54d0 0x7f518810be70 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:47.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.760+0000 7f518759e700 1 --2- 172.21.15.139:0/376536621 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51880f6770 0x7f518810e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:47.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.760+0000 7f518759e700 1 -- 172.21.15.139:0/376536621 >> 172.21.15.139:0/376536621 conn(0x7f51880f0c10 msgr2=0x7f51880fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:47.747 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.760+0000 7f518759e700 1 -- 172.21.15.139:0/376536621 shutdown_connections 2021-11-20T06:07:47.747 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:47.760+0000 7f518759e700 1 -- 172.21.15.139:0/376536621 wait complete. 2021-11-20T06:07:47.747 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 80 2021-11-20T06:07:47.757 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:47.262249+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24513}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24513': {'gid': 24513, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 79, 'state': 'up:reconnect', 'state_seq': 6, 'addr': '172.21.15.178:6825/1490972326', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 1490972326}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 1490972326}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:47.757 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24513:up:reconnect 2021-11-20T06:07:48.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:48 smithi139 conmon[33501]: cluster 2021-11-20T06:07:47.266696+0000 mon.smithi139 (mon.0) 842 : cluster [DBG] mds.? [v2:172.21.15.178:6824/1490972326,v1:172.21.15.178:6825/1490972326] up:reconnect 2021-11-20T06:07:48.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:48 smithi139 conmon[33501]: cluster 2021-11-20T06:07:47.266757+0000 mon.smithi139 (mon.0) 843 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:reconnect} 1 up:standby 2021-11-20T06:07:48.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:48 smithi139 conmon[33501]: cluster 2021-11-20T06:07:47.287485+0000 mds.cephfs.smithi178.blvkbx (mds.0) 1 : cluster [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0209996 2021-11-20T06:07:48.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:48 smithi139 conmon[33501]: audit 2021-11-20T06:07:47.570635+0000 mon.smithi139 (mon.0) 844 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:07:48.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:48 smithi139 conmon[33501]: audit 2021-11-20T06:07:47.759167+0000 mon.smithi139 (mon.0) 845 : audit [DBG] from='client.? 172.21.15.139:0/376536621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:48.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:48 smithi139 conmon[33501]: audit 2021-11 2021-11-20T06:07:48.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:48 smithi139 conmon[33501]: -20T06:07:47.954900+0000 mon.smithi139 (mon.0) 846 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:07:48.687 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:48 smithi178 conmon[36600]: cluster 2021-11-20T06:07:47.266696 2021-11-20T06:07:48.687 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:48 smithi178 conmon[36600]: +0000 mon.smithi139 (mon.0) 842 : cluster [DBG] mds.? [v2:172.21.15.178:6824/1490972326,v1:172.21.15.178:6825/1490972326] up:reconnect 2021-11-20T06:07:48.688 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:48 smithi178 conmon[36600]: cluster 2021-11-20T06:07:47.266757+0000 mon.smithi139 (mon.0) 843 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:reconnect} 1 up:standby 2021-11-20T06:07:48.688 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:48 smithi178 conmon[36600]: cluster 2021-11-20T06:07:47.287485+0000 mds.cephfs.smithi178.blvkbx (mds.0) 1 : cluster [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0209996 2021-11-20T06:07:48.689 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:48 smithi178 conmon[36600]: audit 2021-11-20T06:07:47.570635+0000 mon.smithi139 (mon.0) 844 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-20T06:07:48.689 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:48 smithi178 conmon[36600]: audit 2021-11-20T06:07:47.759167+0000 mon.smithi139 (mon.0) 845 : audit [DBG] from='client.? 172.21.15.139:0/376536621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:48.689 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:48 smithi178 conmon[36600]: audit 2021-11-20T06:07:47.954900+0000 mon.smithi139 (mon.0) 846 : audit [INF] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' 2021-11-20T06:07:48.759 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:48.879 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.890+0000 7fc545f4f700 1 -- 172.21.15.139:0/2431319152 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5400f5300 msgr2=0x7fc5400f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:48.880 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.890+0000 7fc545f4f700 1 --2- 172.21.15.139:0/2431319152 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5400f5300 0x7fc5400f7a80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fc534004660 tx=0x7fc534009b30).stop 2021-11-20T06:07:48.880 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.890+0000 7fc545f4f700 1 -- 172.21.15.139:0/2431319152 shutdown_connections 2021-11-20T06:07:48.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.890+0000 7fc545f4f700 1 --2- 172.21.15.139:0/2431319152 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5400f5300 0x7fc5400f7a80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:48.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.890+0000 7fc545f4f700 1 --2- 172.21.15.139:0/2431319152 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5400f4960 0x7fc5400f4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:07:48.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.890+0000 7fc545f4f700 1 -- 172.21.15.139:0/2431319152 >> 172.21.15.139:0/2431319152 conn(0x7fc5400f0440 msgr2=0x7fc5400f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:48.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.890+0000 7fc545f4f700 1 -- 172.21.15.139:0/2431319152 shutdown_connections 2021-11-20T06:07:48.882 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.890+0000 7fc545f4f700 1 -- 172.21.15.139:0/2431319152 wait complete. 2021-11-20T06:07:48.882 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.890+0000 7fc545f4f700 1 Processor -- start 2021-11-20T06:07:48.882 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc545f4f700 1 -- start start 2021-11-20T06:07:48.883 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc545f4f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5400f4960 0x7fc540106f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:48.883 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc545f4f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5400f5300 0x7fc540107410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:48.883 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc545f4f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc540107a50 con 0x7fc5400f4960 2021-11-20T06:07:48.884 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc545f4f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc540107b90 con 0x7fc5400f5300 2021-11-20T06:07:48.884 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc53f7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5400f4960 0x7fc540106f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:48.884 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc53f7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5400f4960 0x7fc540106f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58952/0 (socket says 172.21.15.139:58952) 2021-11-20T06:07:48.884 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc53f7fe700 1 -- 172.21.15.139:0/901076887 learned_addr learned my addr 172.21.15.139:0/901076887 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:48.885 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc53effd700 1 --2- 172.21.15.139:0/901076887 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5400f5300 0x7fc540107410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:48.885 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc53f7fe700 1 -- 172.21.15.139:0/901076887 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5400f5300 msgr2=0x7fc540107410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:48.885 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc53f7fe700 1 --2- 172.21.15.139:0/901076887 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5400f5300 0x7fc540107410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:48.885 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc53f7fe700 1 -- 172.21.15.139:0/901076887 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc534005040 con 0x7fc5400f4960 2021-11-20T06:07:48.886 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc53effd700 1 --2- 172.21.15.139:0/901076887 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5400f5300 0x7fc540107410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:07:48.886 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc53f7fe700 1 --2- 172.21.15.139:0/901076887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5400f4960 0x7fc540106f20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fc5300093f0 tx=0x7fc530006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:48.886 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc527fff700 1 -- 172.21.15.139:0/901076887 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc530013640 con 0x7fc5400f4960 2021-11-20T06:07:48.886 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc545f4f700 1 -- 172.21.15.139:0/901076887 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54010c630 con 0x7fc5400f4960 2021-11-20T06:07:48.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc527fff700 1 -- 172.21.15.139:0/901076887 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc530009e40 con 0x7fc5400f4960 2021-11-20T06:07:48.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc527fff700 1 -- 172.21.15.139:0/901076887 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc53000dbf0 con 0x7fc5400f4960 2021-11-20T06:07:48.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.891+0000 7fc545f4f700 1 -- 172.21.15.139:0/901076887 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54010cbf0 con 0x7fc5400f4960 2021-11-20T06:07:48.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.892+0000 7fc545f4f700 1 -- 172.21.15.139:0/901076887 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc540027a00 con 0x7fc5400f4960 2021-11-20T06:07:48.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.893+0000 7fc527fff700 1 -- 172.21.15.139:0/901076887 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc53001a060 con 0x7fc5400f4960 2021-11-20T06:07:48.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.893+0000 7fc527fff700 1 --2- 172.21.15.139:0/901076887 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc52805c2a0 0x7fc52805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:48.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.893+0000 7fc527fff700 1 -- 172.21.15.139:0/901076887 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fc530076450 con 0x7fc5400f4960 2021-11-20T06:07:48.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.893+0000 7fc53effd700 1 --2- 172.21.15.139:0/901076887 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc52805c2a0 0x7fc52805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:48.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.893+0000 7fc53effd700 1 --2- 172.21.15.139:0/901076887 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc52805c2a0 0x7fc52805e760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fc534004e30 tx=0x7fc534009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:48.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:48.897+0000 7fc527fff700 1 -- 172.21.15.139:0/901076887 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc530048bb0 con 0x7fc5400f4960 2021-11-20T06:07:49.064 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:49.079+0000 7fc545f4f700 1 -- 172.21.15.139:0/901076887 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5400feef0 con 0x7fc5400f4960 2021-11-20T06:07:49.065 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:49.080+0000 7fc527fff700 1 -- 172.21.15.139:0/901076887 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+2188 (secure 0 0 0) 0x7fc53004c1d0 con 0x7fc5400f4960 2021-11-20T06:07:49.065 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:49.066 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":81,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:48.265866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24513},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24513":{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":79,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:49.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:49.081+0000 7fc545f4f700 1 -- 172.21.15.139:0/901076887 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc52805c2a0 msgr2=0x7fc52805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:49.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:49.081+0000 7fc545f4f700 1 --2- 172.21.15.139:0/901076887 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc52805c2a0 0x7fc52805e760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fc534004e30 tx=0x7fc534009b30).stop 2021-11-20T06:07:49.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:49.081+0000 7fc545f4f700 1 -- 172.21.15.139:0/901076887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5400f4960 msgr2=0x7fc540106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:49.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:49.081+0000 7fc545f4f700 1 --2- 172.21.15.139:0/901076887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5400f4960 0x7fc540106f20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fc5300093f0 tx=0x7fc530006ec0).stop 2021-11-20T06:07:49.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:49.082+0000 7fc545f4f700 1 -- 172.21.15.139:0/901076887 shutdown_connections 2021-11-20T06:07:49.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:49.082+0000 7fc545f4f700 1 --2- 172.21.15.139:0/901076887 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc52805c2a0 0x7fc52805e760 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:49.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:49.082+0000 7fc545f4f700 1 --2- 172.21.15.139:0/901076887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5400f4960 0x7fc540106f20 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:49.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:49.082+0000 7fc545f4f700 1 --2- 172.21.15.139:0/901076887 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5400f5300 0x7fc540107410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:49.069 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:49.082+0000 7fc545f4f700 1 -- 172.21.15.139:0/901076887 >> 172.21.15.139:0/901076887 conn(0x7fc5400f0440 msgr2=0x7fc5400fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:49.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:49.082+0000 7fc545f4f700 1 -- 172.21.15.139:0/901076887 shutdown_connections 2021-11-20T06:07:49.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:49.082+0000 7fc545f4f700 1 -- 172.21.15.139:0/901076887 wait complete. 2021-11-20T06:07:49.071 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 81 2021-11-20T06:07:49.078 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:48.265866+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24513}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24513': {'gid': 24513, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/1490972326', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 1490972326}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 1490972326}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:49.078 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24513:up:rejoin 2021-11-20T06:07:49.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:49 smithi139 conmon[33501]: cluster 2021-11-20T06:07:48.276175+0000 mon.smithi139 (mon.0 2021-11-20T06:07:49.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:49 smithi139 conmon[33501]: ) 847 : cluster [DBG] mds.? [v2:172.21.15.178:6824/1490972326,v1:172.21.15.178:6825/1490972326] up:rejoin 2021-11-20T06:07:49.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:49 smithi139 conmon[33501]: cluster 2021-11-20T06:07:48.276235+0000 mon.smithi139 (mon.0) 848 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:rejoin} 1 up:standby 2021-11-20T06:07:49.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:49 smithi139 conmon[33501]: cluster 2021-11-20 2021-11-20T06:07:49.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:49 smithi139 conmon[33501]: T06:07:48.276350+0000 mgr.smithi139.dplyxn (mgr.14162) 320 : cluster [DBG] 2021-11-20T06:07:49.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:49 smithi139 conmon[33501]: pgmap v296: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T06:07:49.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:49 smithi139 conmon[33501]: audit 2021-11 2021-11-20T06:07:49.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:49 smithi139 conmon[33501]: -20T06:07:49.080812+0000 mon.smithi139 (mon.0) 849 : 2021-11-20T06:07:49.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:49 smithi139 conmon[33501]: audit [DBG] from='client.? 172.21.15.139:0/901076887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:49.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:49 smithi178 conmon[36600]: cluster 2021-11-20T06:07:48.276175+0000 mon.smithi139 (mon.0) 847 : cluster [DBG] mds.? [v2:172.21.15.178:6824/1490972326,v1:172.21.15.178:6825/1490972326] up:rejoin 2021-11-20T06:07:49.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:49 smithi178 conmon[36600]: cluster 2021-11-20T06:07:48.276235+0000 mon.smithi139 (mon.0) 848 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:rejoin} 1 up:standby 2021-11-20T06:07:49.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:49 smithi178 conmon[36600]: cluster 2021-11-20T06:07:48.276350+0000 mgr.smithi139.dplyxn (mgr.14162) 320 : cluster [DBG] pgmap v296: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 0 op/s 2021-11-20T06:07:49.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:49 smithi178 conmon[36600]: audit 2021-11-20T06:07:49.080812+0000 mon.smithi139 (mon.0) 849 : audit [DBG] from='client.? 172.21.15.139:0/901076887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:50.080 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:50.204 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.218+0000 7f99d0b03700 1 -- 172.21.15.139:0/2350650937 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99cc0f5590 msgr2=0x7f99cc0f59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:50.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.218+0000 7f99d0b03700 1 --2- 172.21.15.139:0/2350650937 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99cc0f5590 0x7f99cc0f59b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f99bc0099e0 tx=0x7f99bc004d10).stop 2021-11-20T06:07:50.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.218+0000 7f99d0b03700 1 -- 172.21.15.139:0/2350650937 shutdown_connections 2021-11-20T06:07:50.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.218+0000 7f99d0b03700 1 --2- 172.21.15.139:0/2350650937 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99cc0f6830 0x7f99cc0f6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:50.205 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.218+0000 7f99d0b03700 1 --2- 172.21.15.139:0/2350650937 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99cc0f5590 0x7f99cc0f59b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:50.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.218+0000 7f99d0b03700 1 -- 172.21.15.139:0/2350650937 >> 172.21.15.139:0/2350650937 conn(0x7f99cc0f0cd0 msgr2=0x7f99cc0f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:50.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.218+0000 7f99d0b03700 1 -- 172.21.15.139:0/2350650937 shutdown_connections 2021-11-20T06:07:50.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.218+0000 7f99d0b03700 1 -- 172.21.15.139:0/2350650937 wait complete. 2021-11-20T06:07:50.206 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.218+0000 7f99d0b03700 1 Processor -- start 2021-11-20T06:07:50.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.218+0000 7f99d0b03700 1 -- start start 2021-11-20T06:07:50.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.218+0000 7f99d0b03700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99cc0f5590 0x7f99cc060530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:50.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.218+0000 7f99d0b03700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99cc0f6830 0x7f99cc10bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:50.207 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.218+0000 7f99d0b03700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc10c290 con 0x7f99cc0f6830 2021-11-20T06:07:50.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.218+0000 7f99d0b03700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99cc10c3d0 con 0x7f99cc0f5590 2021-11-20T06:07:50.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.218+0000 7f99c9d9b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99cc0f6830 0x7f99cc10bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:50.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.219+0000 7f99c9d9b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99cc0f6830 0x7f99cc10bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58966/0 (socket says 172.21.15.139:58966) 2021-11-20T06:07:50.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.219+0000 7f99c9d9b700 1 -- 172.21.15.139:0/758331855 learned_addr learned my addr 172.21.15.139:0/758331855 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:50.208 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.219+0000 7f99ca59c700 1 --2- 172.21.15.139:0/758331855 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99cc0f5590 0x7f99cc060530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:50.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.219+0000 7f99c9d9b700 1 -- 172.21.15.139:0/758331855 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99cc0f5590 msgr2=0x7f99cc060530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:50.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.219+0000 7f99c9d9b700 1 --2- 172.21.15.139:0/758331855 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99cc0f5590 0x7f99cc060530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:50.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.219+0000 7f99c9d9b700 1 -- 172.21.15.139:0/758331855 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99bc009640 con 0x7f99cc0f6830 2021-11-20T06:07:50.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.219+0000 7f99c9d9b700 1 --2- 172.21.15.139:0/758331855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99cc0f6830 0x7f99cc10bda0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f99b400c560 tx=0x7f99b4007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:50.209 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.219+0000 7f99c2ffd700 1 -- 172.21.15.139:0/758331855 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99b400f650 con 0x7f99cc0f6830 2021-11-20T06:07:50.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.219+0000 7f99c2ffd700 1 -- 172.21.15.139:0/758331855 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99b400fc60 con 0x7f99cc0f6830 2021-11-20T06:07:50.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.219+0000 7f99d0b03700 1 -- 172.21.15.139:0/758331855 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99cc10c630 con 0x7f99cc0f6830 2021-11-20T06:07:50.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.219+0000 7f99c2ffd700 1 -- 172.21.15.139:0/758331855 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99b4012c10 con 0x7f99cc0f6830 2021-11-20T06:07:50.210 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.219+0000 7f99d0b03700 1 -- 172.21.15.139:0/758331855 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99cc10cb70 con 0x7f99cc0f6830 2021-11-20T06:07:50.211 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.220+0000 7f99d0b03700 1 -- 172.21.15.139:0/758331855 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99cc027a00 con 0x7f99cc0f6830 2021-11-20T06:07:50.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.220+0000 7f99c2ffd700 1 -- 172.21.15.139:0/758331855 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f99b4010070 con 0x7f99cc0f6830 2021-11-20T06:07:50.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.221+0000 7f99c2ffd700 1 --2- 172.21.15.139:0/758331855 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f99b805c2a0 0x7f99b805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:50.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.221+0000 7f99c2ffd700 1 -- 172.21.15.139:0/758331855 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f99b4076030 con 0x7f99cc0f6830 2021-11-20T06:07:50.212 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.221+0000 7f99ca59c700 1 --2- 172.21.15.139:0/758331855 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f99b805c2a0 0x7f99b805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:50.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.222+0000 7f99ca59c700 1 --2- 172.21.15.139:0/758331855 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f99b805c2a0 0x7f99b805e760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f99bc00b620 tx=0x7f99bc004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:50.213 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.225+0000 7f99c2ffd700 1 -- 172.21.15.139:0/758331855 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99b40452b0 con 0x7f99cc0f6830 2021-11-20T06:07:50.403 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.418+0000 7f99d0b03700 1 -- 172.21.15.139:0/758331855 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99cc0fdd20 con 0x7f99cc0f6830 2021-11-20T06:07:50.403 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.419+0000 7f99c2ffd700 1 -- 172.21.15.139:0/758331855 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f99b404c220 con 0x7f99cc0f6830 2021-11-20T06:07:50.404 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:50.405 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":81,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:48.265866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24513},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24513":{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":79,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:50.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.420+0000 7f99d0b03700 1 -- 172.21.15.139:0/758331855 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f99b805c2a0 msgr2=0x7f99b805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:50.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.420+0000 7f99d0b03700 1 --2- 172.21.15.139:0/758331855 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f99b805c2a0 0x7f99b805e760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f99bc00b620 tx=0x7f99bc004d10).stop 2021-11-20T06:07:50.406 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.421+0000 7f99d0b03700 1 -- 172.21.15.139:0/758331855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99cc0f6830 msgr2=0x7f99cc10bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:50.407 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.421+0000 7f99d0b03700 1 --2- 172.21.15.139:0/758331855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99cc0f6830 0x7f99cc10bda0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f99b400c560 tx=0x7f99b4007e20).stop 2021-11-20T06:07:50.407 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.421+0000 7f99d0b03700 1 -- 172.21.15.139:0/758331855 shutdown_connections 2021-11-20T06:07:50.407 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.421+0000 7f99d0b03700 1 --2- 172.21.15.139:0/758331855 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f99b805c2a0 0x7f99b805e760 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:50.407 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.421+0000 7f99d0b03700 1 --2- 172.21.15.139:0/758331855 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99cc0f5590 0x7f99cc060530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:50.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.421+0000 7f99d0b03700 1 --2- 172.21.15.139:0/758331855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f99cc0f6830 0x7f99cc10bda0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:50.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.421+0000 7f99d0b03700 1 -- 172.21.15.139:0/758331855 >> 172.21.15.139:0/758331855 conn(0x7f99cc0f0cd0 msgr2=0x7f99cc0fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:50.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.421+0000 7f99d0b03700 1 -- 172.21.15.139:0/758331855 shutdown_connections 2021-11-20T06:07:50.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:50.421+0000 7f99d0b03700 1 -- 172.21.15.139:0/758331855 wait complete. 2021-11-20T06:07:50.408 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 82 2021-11-20T06:07:50.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:50 smithi139 conmon[33501]: cluster 2021-11-20T06:07:49.282420+0000 mon.smithi139 (mon.0 2021-11-20T06:07:50.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:50 smithi139 conmon[33501]: ) 850 : cluster [DBG] mds.? [v2:172.21.15.178:6826/2304740608,v1:172.21.15.178:6827/2304740608] up:standby 2021-11-20T06:07:50.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:50 smithi139 conmon[33501]: cluster 2021-11-20T06:07:49.282491+0000 2021-11-20T06:07:50.416 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:50 smithi139 conmon[33501]: mon.smithi139 (mon.0) 851 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:rejoin} 1 up:standby 2021-11-20T06:07:50.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:48.265866+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24513}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24513': {'gid': 24513, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/1490972326', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 1490972326}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 1490972326}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:50.418 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24513:up:rejoin 2021-11-20T06:07:50.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:50 smithi178 conmon[36600]: cluster 2021-11-20T06:07:49.282420+0000 mon.smithi139 (mon.0) 850 : cluster 2021-11-20T06:07:50.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:50 smithi178 conmon[36600]: [DBG] mds.? [v2:172.21.15.178:6826/2304740608,v1:172.21.15.178:6827/2304740608] up:standby 2021-11-20T06:07:50.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:50 smithi178 conmon[36600]: cluster 2021-11-20T06:07:49.282491+0000 mon.smithi139 (mon.0) 851 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:rejoin} 1 up:standby 2021-11-20T06:07:51.419 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:51.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:51 smithi139 conmon[33501]: cluster 2021-11-20T06:07:50.276847+0000 mgr.smithi139.dplyxn (mgr.14162) 321 : cluster [DBG] pgmap v297: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:07:51.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:51 smithi139 conmon[33501]: audit 2021-11-20T06:07:50.419840+0000 mon.smithi139 (mon.0) 852 : audit [DBG] from='client.? 172.21.15.139:0/758331855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:51.551 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.561+0000 7fdcb6b60700 1 -- 172.21.15.139:0/2575231688 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb00f6850 msgr2=0x7fdcb00f6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:51.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.561+0000 7fdcb6b60700 1 --2- 172.21.15.139:0/2575231688 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb00f6850 0x7fdcb00f6cb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fdc9c004660 tx=0x7fdc9c009b30).stop 2021-11-20T06:07:51.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.562+0000 7fdcb6b60700 1 -- 172.21.15.139:0/2575231688 shutdown_connections 2021-11-20T06:07:51.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.562+0000 7fdcb6b60700 1 --2- 172.21.15.139:0/2575231688 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb00f6850 0x7fdcb00f6cb0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:51.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.562+0000 7fdcb6b60700 1 --2- 172.21.15.139:0/2575231688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcb00f55b0 0x7fdcb00f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:51.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.562+0000 7fdcb6b60700 1 -- 172.21.15.139:0/2575231688 >> 172.21.15.139:0/2575231688 conn(0x7fdcb00f0d10 msgr2=0x7fdcb00f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:51.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.562+0000 7fdcb6b60700 1 -- 172.21.15.139:0/2575231688 shutdown_connections 2021-11-20T06:07:51.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.562+0000 7fdcb6b60700 1 -- 172.21.15.139:0/2575231688 wait complete. 2021-11-20T06:07:51.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.562+0000 7fdcb6b60700 1 Processor -- start 2021-11-20T06:07:51.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.562+0000 7fdcb6b60700 1 -- start start 2021-11-20T06:07:51.556 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.562+0000 7fdcb6b60700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb00f55b0 0x7fdcb0105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:51.556 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.562+0000 7fdcb6b60700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcb00f6850 0x7fdcb0106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:51.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.562+0000 7fdcb6b60700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb0106890 con 0x7fdcb00f55b0 2021-11-20T06:07:51.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.562+0000 7fdcb6b60700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb01069d0 con 0x7fdcb00f6850 2021-11-20T06:07:51.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.563+0000 7fdcb48fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb00f55b0 0x7fdcb0105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:51.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.563+0000 7fdcb48fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb00f55b0 0x7fdcb0105da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58976/0 (socket says 172.21.15.139:58976) 2021-11-20T06:07:51.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.563+0000 7fdcb48fc700 1 -- 172.21.15.139:0/2300563417 learned_addr learned my addr 172.21.15.139:0/2300563417 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:51.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.563+0000 7fdcaffff700 1 --2- 172.21.15.139:0/2300563417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcb00f6850 0x7fdcb0106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:51.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.563+0000 7fdcb48fc700 1 -- 172.21.15.139:0/2300563417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcb00f6850 msgr2=0x7fdcb0106290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:51.560 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.563+0000 7fdcb48fc700 1 --2- 172.21.15.139:0/2300563417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcb00f6850 0x7fdcb0106290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:51.560 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.563+0000 7fdcb48fc700 1 -- 172.21.15.139:0/2300563417 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c005040 con 0x7fdcb00f55b0 2021-11-20T06:07:51.561 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.563+0000 7fdcb48fc700 1 --2- 172.21.15.139:0/2300563417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb00f55b0 0x7fdcb0105da0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fdca400e9a0 tx=0x7fdca4007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:51.561 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.563+0000 7fdcad7fa700 1 -- 172.21.15.139:0/2300563417 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdca400c9b0 con 0x7fdcb00f55b0 2021-11-20T06:07:51.562 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.563+0000 7fdcad7fa700 1 -- 172.21.15.139:0/2300563417 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdca4011470 con 0x7fdcb00f55b0 2021-11-20T06:07:51.562 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.563+0000 7fdcad7fa700 1 -- 172.21.15.139:0/2300563417 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdca40105f0 con 0x7fdcb00f55b0 2021-11-20T06:07:51.563 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.563+0000 7fdcb6b60700 1 -- 172.21.15.139:0/2300563417 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb005fac0 con 0x7fdcb00f55b0 2021-11-20T06:07:51.563 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.563+0000 7fdcb6b60700 1 -- 172.21.15.139:0/2300563417 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb0060080 con 0x7fdcb00f55b0 2021-11-20T06:07:51.563 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.564+0000 7fdcb6b60700 1 -- 172.21.15.139:0/2300563417 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb00fdd40 con 0x7fdcb00f55b0 2021-11-20T06:07:51.564 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.564+0000 7fdcad7fa700 1 -- 172.21.15.139:0/2300563417 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fdca40115e0 con 0x7fdcb00f55b0 2021-11-20T06:07:51.564 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.565+0000 7fdcad7fa700 1 --2- 172.21.15.139:0/2300563417 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdca005c2a0 0x7fdca005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:51.565 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.565+0000 7fdcad7fa700 1 -- 172.21.15.139:0/2300563417 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fdca40792c0 con 0x7fdcb00f55b0 2021-11-20T06:07:51.565 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.565+0000 7fdcaffff700 1 --2- 172.21.15.139:0/2300563417 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdca005c2a0 0x7fdca005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:51.566 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.565+0000 7fdcaffff700 1 --2- 172.21.15.139:0/2300563417 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdca005c2a0 0x7fdca005e760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fdc9c009d60 tx=0x7fdc9c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:51.567 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.569+0000 7fdcad7fa700 1 -- 172.21.15.139:0/2300563417 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdca404bb00 con 0x7fdcb00f55b0 2021-11-20T06:07:51.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:51 smithi178 conmon[36600]: cluster 2021-11-20T06:07:50.276847+0000 mgr.smithi139.dplyxn (mgr.14162) 321 2021-11-20T06:07:51.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:51 smithi178 conmon[36600]: : cluster [DBG] pgmap v297: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:07:51.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:51 smithi178 conmon[36600]: audit 2021-11-20T06:07:50.419840+0000 mon.smithi139 (mon.0) 852 : audit [DBG] from='client.? 172.21.15.139:0/758331855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:51.736 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.752+0000 7fdcb6b60700 1 -- 172.21.15.139:0/2300563417 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcb010b380 con 0x7fdcb00f55b0 2021-11-20T06:07:51.737 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.753+0000 7fdcad7fa700 1 -- 172.21.15.139:0/2300563417 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2187 (secure 0 0 0) 0x7fdca404f120 con 0x7fdcb00f55b0 2021-11-20T06:07:51.737 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:51.738 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":81,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:48.265866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24513},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24513":{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":79,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:51.739 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.754+0000 7fdcb6b60700 1 -- 172.21.15.139:0/2300563417 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdca005c2a0 msgr2=0x7fdca005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:51.740 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.754+0000 7fdcb6b60700 1 --2- 172.21.15.139:0/2300563417 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdca005c2a0 0x7fdca005e760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fdc9c009d60 tx=0x7fdc9c009b30).stop 2021-11-20T06:07:51.740 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.754+0000 7fdcb6b60700 1 -- 172.21.15.139:0/2300563417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb00f55b0 msgr2=0x7fdcb0105da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:51.740 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.754+0000 7fdcb6b60700 1 --2- 172.21.15.139:0/2300563417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb00f55b0 0x7fdcb0105da0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fdca400e9a0 tx=0x7fdca4007e20).stop 2021-11-20T06:07:51.740 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.754+0000 7fdcb6b60700 1 -- 172.21.15.139:0/2300563417 shutdown_connections 2021-11-20T06:07:51.741 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.754+0000 7fdcb6b60700 1 --2- 172.21.15.139:0/2300563417 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fdca005c2a0 0x7fdca005e760 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:51.741 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.754+0000 7fdcb6b60700 1 --2- 172.21.15.139:0/2300563417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fdcb00f55b0 0x7fdcb0105da0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:51.741 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.754+0000 7fdcb6b60700 1 --2- 172.21.15.139:0/2300563417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcb00f6850 0x7fdcb0106290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:51.741 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.754+0000 7fdcb6b60700 1 -- 172.21.15.139:0/2300563417 >> 172.21.15.139:0/2300563417 conn(0x7fdcb00f0d10 msgr2=0x7fdcb00fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:51.742 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.754+0000 7fdcb6b60700 1 -- 172.21.15.139:0/2300563417 shutdown_connections 2021-11-20T06:07:51.742 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:51.754+0000 7fdcb6b60700 1 -- 172.21.15.139:0/2300563417 wait complete. 2021-11-20T06:07:51.742 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 82 2021-11-20T06:07:51.751 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:48.265866+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24513}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24513': {'gid': 24513, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/1490972326', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 1490972326}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 1490972326}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:51.751 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24513:up:rejoin 2021-11-20T06:07:52.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:52 smithi178 conmon[36600]: audit 2021-11-20T06:07:51.753398+0000 mon.smithi139 (mon.0) 853 2021-11-20T06:07:52.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:52 smithi178 conmon[36600]: : audit [DBG] from='client.? 172.21.15.139:0/2300563417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:52.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:52 smithi139 conmon[33501]: audit 2021-11-20T06:07:51.753398+0000 mon.smithi139 (mon.0) 853 : audit [DBG] from='client.? 172.21.15.139:0/2300563417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:52.753 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:52.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.894+0000 7f9cf5017700 1 -- 172.21.15.139:0/2391477180 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cf0060100 msgr2=0x7f9cf00f9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:52.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.894+0000 7f9cf5017700 1 --2- 172.21.15.139:0/2391477180 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cf0060100 0x7f9cf00f9310 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f9ce00099e0 tx=0x7f9ce0004d10).stop 2021-11-20T06:07:52.881 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.894+0000 7f9cf5017700 1 -- 172.21.15.139:0/2391477180 shutdown_connections 2021-11-20T06:07:52.882 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.894+0000 7f9cf5017700 1 --2- 172.21.15.139:0/2391477180 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9cf00f9890 0x7f9cf00fbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:52.882 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.894+0000 7f9cf5017700 1 --2- 172.21.15.139:0/2391477180 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cf0060100 0x7f9cf00f9310 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:52.882 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.894+0000 7f9cf5017700 1 -- 172.21.15.139:0/2391477180 >> 172.21.15.139:0/2391477180 conn(0x7f9cf00f0d10 msgr2=0x7f9cf00f3150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:52.882 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.894+0000 7f9cf5017700 1 -- 172.21.15.139:0/2391477180 shutdown_connections 2021-11-20T06:07:52.883 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.894+0000 7f9cf5017700 1 -- 172.21.15.139:0/2391477180 wait complete. 2021-11-20T06:07:52.883 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.894+0000 7f9cf5017700 1 Processor -- start 2021-11-20T06:07:52.883 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.894+0000 7f9cf5017700 1 -- start start 2021-11-20T06:07:52.883 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cf5017700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9cf0060100 0x7f9cf0106fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:52.883 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cf5017700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cf00f9890 0x7f9cf01074b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:52.884 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cf5017700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf0107a60 con 0x7f9cf00f9890 2021-11-20T06:07:52.884 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cf5017700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf0107ba0 con 0x7f9cf0060100 2021-11-20T06:07:52.884 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cee59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cf00f9890 0x7f9cf01074b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:52.884 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cee59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cf00f9890 0x7f9cf01074b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:58990/0 (socket says 172.21.15.139:58990) 2021-11-20T06:07:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cee59c700 1 -- 172.21.15.139:0/1202410994 learned_addr learned my addr 172.21.15.139:0/1202410994 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9ceed9d700 1 --2- 172.21.15.139:0/1202410994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9cf0060100 0x7f9cf0106fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cee59c700 1 -- 172.21.15.139:0/1202410994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9cf0060100 msgr2=0x7f9cf0106fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cee59c700 1 --2- 172.21.15.139:0/1202410994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9cf0060100 0x7f9cf0106fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:52.885 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cee59c700 1 -- 172.21.15.139:0/1202410994 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce0009640 con 0x7f9cf00f9890 2021-11-20T06:07:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cee59c700 1 --2- 172.21.15.139:0/1202410994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cf00f9890 0x7f9cf01074b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f9ce400c560 tx=0x7f9ce4007dd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cd77fe700 1 -- 172.21.15.139:0/1202410994 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ce400f650 con 0x7f9cf00f9890 2021-11-20T06:07:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cf5017700 1 -- 172.21.15.139:0/1202410994 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf010c640 con 0x7f9cf00f9890 2021-11-20T06:07:52.886 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cd77fe700 1 -- 172.21.15.139:0/1202410994 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ce400fc60 con 0x7f9cf00f9890 2021-11-20T06:07:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cd77fe700 1 -- 172.21.15.139:0/1202410994 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ce4012c00 con 0x7f9cf00f9890 2021-11-20T06:07:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.895+0000 7f9cf5017700 1 -- 172.21.15.139:0/1202410994 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf010cb80 con 0x7f9cf00f9890 2021-11-20T06:07:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.896+0000 7f9cf5017700 1 -- 172.21.15.139:0/1202410994 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf0027a00 con 0x7f9cf00f9890 2021-11-20T06:07:52.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.897+0000 7f9cd77fe700 1 -- 172.21.15.139:0/1202410994 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9ce4010070 con 0x7f9cf00f9890 2021-11-20T06:07:52.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.897+0000 7f9cd77fe700 1 --2- 172.21.15.139:0/1202410994 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9cd805c2a0 0x7f9cd805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:52.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.897+0000 7f9cd77fe700 1 -- 172.21.15.139:0/1202410994 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f9ce4076130 con 0x7f9cf00f9890 2021-11-20T06:07:52.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.897+0000 7f9ceed9d700 1 --2- 172.21.15.139:0/1202410994 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9cd805c2a0 0x7f9cd805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:52.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.897+0000 7f9ceed9d700 1 --2- 172.21.15.139:0/1202410994 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9cd805c2a0 0x7f9cd805e760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f9ce0003380 tx=0x7f9ce0004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:52.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:52.901+0000 7f9cd77fe700 1 -- 172.21.15.139:0/1202410994 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ce4048890 con 0x7f9cf00f9890 2021-11-20T06:07:53.067 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:53.083+0000 7f9cf5017700 1 -- 172.21.15.139:0/1202410994 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cf00feea0 con 0x7f9cf00f9890 2021-11-20T06:07:53.068 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:53.084+0000 7f9cd77fe700 1 -- 172.21.15.139:0/1202410994 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f9ce404beb0 con 0x7f9cf00f9890 2021-11-20T06:07:53.069 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:53.069 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":81,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:48.265866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24513},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24513":{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":79,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:53.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:53.085+0000 7f9cf5017700 1 -- 172.21.15.139:0/1202410994 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9cd805c2a0 msgr2=0x7f9cd805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:53.070 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:53.085+0000 7f9cf5017700 1 --2- 172.21.15.139:0/1202410994 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9cd805c2a0 0x7f9cd805e760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f9ce0003380 tx=0x7f9ce0004d10).stop 2021-11-20T06:07:53.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:53.085+0000 7f9cf5017700 1 -- 172.21.15.139:0/1202410994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cf00f9890 msgr2=0x7f9cf01074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:53.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:53.085+0000 7f9cf5017700 1 --2- 172.21.15.139:0/1202410994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cf00f9890 0x7f9cf01074b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f9ce400c560 tx=0x7f9ce4007dd0).stop 2021-11-20T06:07:53.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:53.085+0000 7f9cf5017700 1 -- 172.21.15.139:0/1202410994 shutdown_connections 2021-11-20T06:07:53.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:53.085+0000 7f9cf5017700 1 --2- 172.21.15.139:0/1202410994 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9cd805c2a0 0x7f9cd805e760 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:53.071 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:53.085+0000 7f9cf5017700 1 --2- 172.21.15.139:0/1202410994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9cf0060100 0x7f9cf0106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:53.072 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:53.085+0000 7f9cf5017700 1 --2- 172.21.15.139:0/1202410994 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9cf00f9890 0x7f9cf01074b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:53.072 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:53.085+0000 7f9cf5017700 1 -- 172.21.15.139:0/1202410994 >> 172.21.15.139:0/1202410994 conn(0x7f9cf00f0d10 msgr2=0x7f9cf00f3150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:53.072 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:53.085+0000 7f9cf5017700 1 -- 172.21.15.139:0/1202410994 shutdown_connections 2021-11-20T06:07:53.072 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:53.085+0000 7f9cf5017700 1 -- 172.21.15.139:0/1202410994 wait complete. 2021-11-20T06:07:53.073 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 82 2021-11-20T06:07:53.082 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:48.265866+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24513}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24513': {'gid': 24513, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/1490972326', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 1490972326}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 1490972326}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:53.082 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24513:up:rejoin 2021-11-20T06:07:53.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:53 smithi139 conmon[33501]: cluster 2021-11-20T06:07:52.277381+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:07:53.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:53 smithi139 conmon[33501]: 322 : cluster [DBG] pgmap v298: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:53.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:53 smithi139 conmon[33501]: audit 2021-11-20T06:07:53.084437+0000 mon.smithi139 (mon.0) 854 : audit [DBG] from='client.? 172.21.15.139:0/1202410994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:53.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:53 smithi178 conmon[36600]: cluster 2021-11-20T06:07:52.277381+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:07:53.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:53 smithi178 conmon[36600]: ) 322 : cluster [DBG] pgmap v298: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:53.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:53 smithi178 conmon[36600]: audit 2021-11-20T06:07:53.084437+0000 mon.smithi139 (mon.0) 854 : audit [DBG] from='client.? 172.21.15.139:0/1202410994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:54.084 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:54.218 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.231+0000 7f5ad3317700 1 -- 172.21.15.139:0/4269670743 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5acc0f5280 msgr2=0x7f5acc0f7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:54.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.231+0000 7f5ad3317700 1 --2- 172.21.15.139:0/4269670743 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5acc0f5280 0x7f5acc0f7bc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f5abc004660 tx=0x7f5abc009b30).stop 2021-11-20T06:07:54.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.231+0000 7f5ad3317700 1 -- 172.21.15.139:0/4269670743 shutdown_connections 2021-11-20T06:07:54.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.231+0000 7f5ad3317700 1 --2- 172.21.15.139:0/4269670743 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5acc0f5280 0x7f5acc0f7bc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:54.219 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.231+0000 7f5ad3317700 1 --2- 172.21.15.139:0/4269670743 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5acc0f48e0 0x7f5acc0f4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:54.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.231+0000 7f5ad3317700 1 -- 172.21.15.139:0/4269670743 >> 172.21.15.139:0/4269670743 conn(0x7f5acc0f0360 msgr2=0x7f5acc0f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:54.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.231+0000 7f5ad3317700 1 -- 172.21.15.139:0/4269670743 shutdown_connections 2021-11-20T06:07:54.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.231+0000 7f5ad3317700 1 -- 172.21.15.139:0/4269670743 wait complete. 2021-11-20T06:07:54.220 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.232+0000 7f5ad3317700 1 Processor -- start 2021-11-20T06:07:54.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.232+0000 7f5ad3317700 1 -- start start 2021-11-20T06:07:54.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.232+0000 7f5ad3317700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5acc0f48e0 0x7f5acc106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:54.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.232+0000 7f5ad3317700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5acc0f5280 0x7f5acc1074a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:54.221 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.232+0000 7f5ad3317700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc107a50 con 0x7f5acc0f5280 2021-11-20T06:07:54.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.232+0000 7f5ad3317700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc107b90 con 0x7f5acc0f48e0 2021-11-20T06:07:54.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.232+0000 7f5ad1b14700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5acc0f5280 0x7f5acc1074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:54.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.232+0000 7f5ad1b14700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5acc0f5280 0x7f5acc1074a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59000/0 (socket says 172.21.15.139:59000) 2021-11-20T06:07:54.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.232+0000 7f5ad1b14700 1 -- 172.21.15.139:0/208611035 learned_addr learned my addr 172.21.15.139:0/208611035 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:54.222 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.232+0000 7f5ad2315700 1 --2- 172.21.15.139:0/208611035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5acc0f48e0 0x7f5acc106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:54.223 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.232+0000 7f5ad1b14700 1 -- 172.21.15.139:0/208611035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5acc0f48e0 msgr2=0x7f5acc106fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:54.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.232+0000 7f5ad1b14700 1 --2- 172.21.15.139:0/208611035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5acc0f48e0 0x7f5acc106fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:54.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.232+0000 7f5ad1b14700 1 -- 172.21.15.139:0/208611035 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5abc005040 con 0x7f5acc0f5280 2021-11-20T06:07:54.224 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.233+0000 7f5ad1b14700 1 --2- 172.21.15.139:0/208611035 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5acc0f5280 0x7f5acc1074a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f5abc009f50 tx=0x7f5abc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:54.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.233+0000 7f5ac2ffd700 1 -- 172.21.15.139:0/208611035 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5abc018070 con 0x7f5acc0f5280 2021-11-20T06:07:54.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.233+0000 7f5ac2ffd700 1 -- 172.21.15.139:0/208611035 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5abc013410 con 0x7f5acc0f5280 2021-11-20T06:07:54.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.233+0000 7f5ad3317700 1 -- 172.21.15.139:0/208611035 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5acc1084d0 con 0x7f5acc0f5280 2021-11-20T06:07:54.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.233+0000 7f5ac2ffd700 1 -- 172.21.15.139:0/208611035 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5abc01c620 con 0x7f5acc0f5280 2021-11-20T06:07:54.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.233+0000 7f5ad3317700 1 -- 172.21.15.139:0/208611035 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5acc108a30 con 0x7f5acc0f5280 2021-11-20T06:07:54.226 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.234+0000 7f5ad3317700 1 -- 172.21.15.139:0/208611035 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5acc045ad0 con 0x7f5acc0f5280 2021-11-20T06:07:54.226 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.234+0000 7f5ac2ffd700 1 -- 172.21.15.139:0/208611035 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5abc01d420 con 0x7f5acc0f5280 2021-11-20T06:07:54.227 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.234+0000 7f5ac2ffd700 1 --2- 172.21.15.139:0/208611035 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f5ab805c300 0x7f5ab805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:54.228 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.235+0000 7f5ac2ffd700 1 -- 172.21.15.139:0/208611035 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f5abc0781b0 con 0x7f5acc0f5280 2021-11-20T06:07:54.228 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.235+0000 7f5ad2315700 1 --2- 172.21.15.139:0/208611035 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f5ab805c300 0x7f5ab805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:54.228 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.235+0000 7f5ad2315700 1 --2- 172.21.15.139:0/208611035 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f5ab805c300 0x7f5ab805e7c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f5ac80041d0 tx=0x7f5ac8009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:54.229 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.238+0000 7f5ac2ffd700 1 -- 172.21.15.139:0/208611035 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5abc04a9f0 con 0x7f5acc0f5280 2021-11-20T06:07:54.408 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.423+0000 7f5ad3317700 1 -- 172.21.15.139:0/208611035 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5acc05f5a0 con 0x7f5acc0f5280 2021-11-20T06:07:54.409 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.423+0000 7f5ac2ffd700 1 -- 172.21.15.139:0/208611035 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f5abc04e010 con 0x7f5acc0f5280 2021-11-20T06:07:54.409 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:54.410 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":81,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:48.265866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24513},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24513":{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":79,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:54.411 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.425+0000 7f5ad3317700 1 -- 172.21.15.139:0/208611035 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f5ab805c300 msgr2=0x7f5ab805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:54.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.425+0000 7f5ad3317700 1 --2- 172.21.15.139:0/208611035 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f5ab805c300 0x7f5ab805e7c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f5ac80041d0 tx=0x7f5ac8009040).stop 2021-11-20T06:07:54.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.425+0000 7f5ad3317700 1 -- 172.21.15.139:0/208611035 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5acc0f5280 msgr2=0x7f5acc1074a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:54.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.425+0000 7f5ad3317700 1 --2- 172.21.15.139:0/208611035 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5acc0f5280 0x7f5acc1074a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f5abc009f50 tx=0x7f5abc009b30).stop 2021-11-20T06:07:54.412 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.425+0000 7f5ad3317700 1 -- 172.21.15.139:0/208611035 shutdown_connections 2021-11-20T06:07:54.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.425+0000 7f5ad3317700 1 --2- 172.21.15.139:0/208611035 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f5ab805c300 0x7f5ab805e7c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:54.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.425+0000 7f5ad3317700 1 --2- 172.21.15.139:0/208611035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5acc0f48e0 0x7f5acc106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:54.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.425+0000 7f5ad3317700 1 --2- 172.21.15.139:0/208611035 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5acc0f5280 0x7f5acc1074a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:54.413 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.425+0000 7f5ad3317700 1 -- 172.21.15.139:0/208611035 >> 172.21.15.139:0/208611035 conn(0x7f5acc0f0360 msgr2=0x7f5acc0fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:54.414 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.425+0000 7f5ad3317700 1 -- 172.21.15.139:0/208611035 shutdown_connections 2021-11-20T06:07:54.414 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:54.425+0000 7f5ad3317700 1 -- 172.21.15.139:0/208611035 wait complete. 2021-11-20T06:07:54.414 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 82 2021-11-20T06:07:54.423 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:48.265866+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24513}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24513': {'gid': 24513, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/1490972326', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 1490972326}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 1490972326}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:54.423 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24513:up:rejoin 2021-11-20T06:07:55.424 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:55.552 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.565+0000 7f8b90086700 1 -- 172.21.15.139:0/3037699392 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b880f7600 msgr2=0x7f8b880f9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:55.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.565+0000 7f8b90086700 1 --2- 172.21.15.139:0/3037699392 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b880f7600 0x7f8b880f9a90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f8b78004660 tx=0x7f8b78009b30).stop 2021-11-20T06:07:55.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.565+0000 7f8b90086700 1 -- 172.21.15.139:0/3037699392 shutdown_connections 2021-11-20T06:07:55.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.565+0000 7f8b90086700 1 --2- 172.21.15.139:0/3037699392 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b880f7600 0x7f8b880f9a90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:55.553 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.565+0000 7f8b90086700 1 --2- 172.21.15.139:0/3037699392 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b88060750 0x7f8b880f7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:55.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.565+0000 7f8b90086700 1 -- 172.21.15.139:0/3037699392 >> 172.21.15.139:0/3037699392 conn(0x7f8b880f0d10 msgr2=0x7f8b880f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:55.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.565+0000 7f8b90086700 1 -- 172.21.15.139:0/3037699392 shutdown_connections 2021-11-20T06:07:55.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.565+0000 7f8b90086700 1 -- 172.21.15.139:0/3037699392 wait complete. 2021-11-20T06:07:55.554 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.565+0000 7f8b90086700 1 Processor -- start 2021-11-20T06:07:55.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.565+0000 7f8b90086700 1 -- start start 2021-11-20T06:07:55.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.565+0000 7f8b90086700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b88060750 0x7f8b8810d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:55.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.565+0000 7f8b90086700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b880f7600 0x7f8b8810d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:55.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.565+0000 7f8b90086700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8810dee0 con 0x7f8b88060750 2021-11-20T06:07:55.555 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.566+0000 7f8b90086700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8810e020 con 0x7f8b880f7600 2021-11-20T06:07:55.556 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.566+0000 7f8b8de22700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b88060750 0x7f8b8810d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:55.556 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.566+0000 7f8b8de22700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b88060750 0x7f8b8810d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59006/0 (socket says 172.21.15.139:59006) 2021-11-20T06:07:55.556 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.566+0000 7f8b8de22700 1 -- 172.21.15.139:0/526867996 learned_addr learned my addr 172.21.15.139:0/526867996 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:55.556 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.566+0000 7f8b8d621700 1 --2- 172.21.15.139:0/526867996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b880f7600 0x7f8b8810d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:55.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.566+0000 7f8b8de22700 1 -- 172.21.15.139:0/526867996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b880f7600 msgr2=0x7f8b8810d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:55.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.566+0000 7f8b8de22700 1 --2- 172.21.15.139:0/526867996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b880f7600 0x7f8b8810d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:55.557 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.566+0000 7f8b8de22700 1 -- 172.21.15.139:0/526867996 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b78005040 con 0x7f8b88060750 2021-11-20T06:07:55.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.566+0000 7f8b8de22700 1 --2- 172.21.15.139:0/526867996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b88060750 0x7f8b8810d440 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f8b8400c760 tx=0x7f8b84004d20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:55.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.566+0000 7f8b7e7fc700 1 -- 172.21.15.139:0/526867996 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b84004200 con 0x7f8b88060750 2021-11-20T06:07:55.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.566+0000 7f8b7e7fc700 1 -- 172.21.15.139:0/526867996 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b840109c0 con 0x7f8b88060750 2021-11-20T06:07:55.558 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.566+0000 7f8b90086700 1 -- 172.21.15.139:0/526867996 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b88112ac0 con 0x7f8b88060750 2021-11-20T06:07:55.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.566+0000 7f8b7e7fc700 1 -- 172.21.15.139:0/526867996 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b84003940 con 0x7f8b88060750 2021-11-20T06:07:55.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.566+0000 7f8b90086700 1 -- 172.21.15.139:0/526867996 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b88113000 con 0x7f8b88060750 2021-11-20T06:07:55.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.567+0000 7f8b90086700 1 -- 172.21.15.139:0/526867996 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b88027a00 con 0x7f8b88060750 2021-11-20T06:07:55.561 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.568+0000 7f8b7e7fc700 1 -- 172.21.15.139:0/526867996 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8b84018020 con 0x7f8b88060750 2021-11-20T06:07:55.561 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.568+0000 7f8b7e7fc700 1 --2- 172.21.15.139:0/526867996 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8b7405c2a0 0x7f8b7405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:55.562 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.568+0000 7f8b7e7fc700 1 -- 172.21.15.139:0/526867996 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f8b84076110 con 0x7f8b88060750 2021-11-20T06:07:55.562 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.568+0000 7f8b8d621700 1 --2- 172.21.15.139:0/526867996 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8b7405c2a0 0x7f8b7405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:55.562 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.568+0000 7f8b8d621700 1 --2- 172.21.15.139:0/526867996 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8b7405c2a0 0x7f8b7405e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f8b78004630 tx=0x7f8b78009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:55.562 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.572+0000 7f8b7e7fc700 1 -- 172.21.15.139:0/526867996 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b840462b0 con 0x7f8b88060750 2021-11-20T06:07:55.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:55 smithi178 conmon[36600]: cluster 2021-11-20T06:07:54.277781+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:07:55.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:55 smithi178 conmon[36600]: 323 : cluster [DBG] pgmap v299: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:55.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:55 smithi178 conmon[36600]: audit 2021-11-20T06:07:54.425332+0000 mon.smithi139 (mon.0) 855 : audit [DBG] from='client.? 172.21.15.139:0/208611035' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:55.741 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.755+0000 7f8b90086700 1 -- 172.21.15.139:0/526867996 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b88105430 con 0x7f8b88060750 2021-11-20T06:07:55.741 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.756+0000 7f8b7e7fc700 1 -- 172.21.15.139:0/526867996 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f8b840488d0 con 0x7f8b88060750 2021-11-20T06:07:55.742 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:55.742 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":81,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:48.265866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24513},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24513":{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":79,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:55.744 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.758+0000 7f8b90086700 1 -- 172.21.15.139:0/526867996 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8b7405c2a0 msgr2=0x7f8b7405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:55.744 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.758+0000 7f8b90086700 1 --2- 172.21.15.139:0/526867996 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8b7405c2a0 0x7f8b7405e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f8b78004630 tx=0x7f8b78009b30).stop 2021-11-20T06:07:55.744 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.758+0000 7f8b90086700 1 -- 172.21.15.139:0/526867996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b88060750 msgr2=0x7f8b8810d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:55.744 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.758+0000 7f8b90086700 1 --2- 172.21.15.139:0/526867996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b88060750 0x7f8b8810d440 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f8b8400c760 tx=0x7f8b84004d20).stop 2021-11-20T06:07:55.745 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.758+0000 7f8b90086700 1 -- 172.21.15.139:0/526867996 shutdown_connections 2021-11-20T06:07:55.745 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.758+0000 7f8b90086700 1 --2- 172.21.15.139:0/526867996 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8b7405c2a0 0x7f8b7405e760 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:55.745 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.758+0000 7f8b90086700 1 --2- 172.21.15.139:0/526867996 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8b88060750 0x7f8b8810d440 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:55.745 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.758+0000 7f8b90086700 1 --2- 172.21.15.139:0/526867996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b880f7600 0x7f8b8810d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:55.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.758+0000 7f8b90086700 1 -- 172.21.15.139:0/526867996 >> 172.21.15.139:0/526867996 conn(0x7f8b880f0d10 msgr2=0x7f8b880f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:55.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.758+0000 7f8b90086700 1 -- 172.21.15.139:0/526867996 shutdown_connections 2021-11-20T06:07:55.746 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:55.758+0000 7f8b90086700 1 -- 172.21.15.139:0/526867996 wait complete. 2021-11-20T06:07:55.746 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 82 2021-11-20T06:07:55.754 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:55 smithi139 conmon[33501]: cluster 2021-11-20T06:07:54.277781+0000 mgr.smithi139.dplyxn (mgr.14162) 323 : cluster [DBG] pgmap v299: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:07:55.754 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:55 smithi139 conmon[33501]: 2021-11-20T06:07:55.754 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:55 smithi139 conmon[33501]: audit 2021-11-20T06:07:54.425332+0000 mon.smithi139 (mon.0) 855 : audit [DBG] from='client.? 172.21.15.139:0/208611035' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:55.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:48.265866+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24513}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24513': {'gid': 24513, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/1490972326', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 1490972326}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 1490972326}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:55.756 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24513:up:rejoin 2021-11-20T06:07:56.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:56 smithi139 conmon[33501]: audit 2021-11-20T06:07:55.757893+0000 mon.smithi139 (mon.0) 856 2021-11-20T06:07:56.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:56 smithi139 conmon[33501]: : audit [DBG] from='client.? 172.21.15.139:0/526867996' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:56.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:56 smithi178 conmon[36600]: audit 2021-11-20T06:07:55.757893+0000 mon.smithi139 ( 2021-11-20T06:07:56.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:56 smithi178 conmon[36600]: mon.0) 856 : audit [DBG] from='client.? 172.21.15.139:0/526867996' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:56.757 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:56.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.898+0000 7f4605e87700 1 -- 172.21.15.139:0/2954901760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46000f6680 msgr2=0x7f46000f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:56.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.898+0000 7f4605e87700 1 --2- 172.21.15.139:0/2954901760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46000f6680 0x7f46000f6aa0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f45f0004660 tx=0x7f45f0009b30).stop 2021-11-20T06:07:56.887 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.898+0000 7f4605e87700 1 -- 172.21.15.139:0/2954901760 shutdown_connections 2021-11-20T06:07:56.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.898+0000 7f4605e87700 1 --2- 172.21.15.139:0/2954901760 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46000f7920 0x7f46000f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:56.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.898+0000 7f4605e87700 1 --2- 172.21.15.139:0/2954901760 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46000f6680 0x7f46000f6aa0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:56.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.898+0000 7f4605e87700 1 -- 172.21.15.139:0/2954901760 >> 172.21.15.139:0/2954901760 conn(0x7f46000f1de0 msgr2=0x7f46000f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:56.888 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.898+0000 7f4605e87700 1 -- 172.21.15.139:0/2954901760 shutdown_connections 2021-11-20T06:07:56.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.898+0000 7f4605e87700 1 -- 172.21.15.139:0/2954901760 wait complete. 2021-11-20T06:07:56.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f4605e87700 1 Processor -- start 2021-11-20T06:07:56.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f4605e87700 1 -- start start 2021-11-20T06:07:56.889 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f4605e87700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46000f6680 0x7f4600106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:56.890 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f4605e87700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46000f7920 0x7f46001073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:56.890 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f4605e87700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4600107950 con 0x7f46000f7920 2021-11-20T06:07:56.890 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f4605e87700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4600107a90 con 0x7f46000f6680 2021-11-20T06:07:56.890 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f45f7fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46000f7920 0x7f46001073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:56.891 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f45f7fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46000f7920 0x7f46001073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59018/0 (socket says 172.21.15.139:59018) 2021-11-20T06:07:56.891 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f45f7fff700 1 -- 172.21.15.139:0/650587162 learned_addr learned my addr 172.21.15.139:0/650587162 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:56.891 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f45ff7fe700 1 --2- 172.21.15.139:0/650587162 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46000f6680 0x7f4600106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:56.891 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f45f7fff700 1 -- 172.21.15.139:0/650587162 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46000f6680 msgr2=0x7f4600106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:56.891 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f45f7fff700 1 --2- 172.21.15.139:0/650587162 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46000f6680 0x7f4600106eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:56.892 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f45f7fff700 1 -- 172.21.15.139:0/650587162 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f0005040 con 0x7f46000f7920 2021-11-20T06:07:56.892 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f45f7fff700 1 --2- 172.21.15.139:0/650587162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46000f7920 0x7f46001073a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f45e80026d0 tx=0x7f45e8009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:56.892 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f45fcff9700 1 -- 172.21.15.139:0/650587162 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45e8009320 con 0x7f46000f7920 2021-11-20T06:07:56.892 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f45fcff9700 1 -- 172.21.15.139:0/650587162 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45e800f040 con 0x7f46000f7920 2021-11-20T06:07:56.893 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f4605e87700 1 -- 172.21.15.139:0/650587162 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f460010c530 con 0x7f46000f7920 2021-11-20T06:07:56.893 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.899+0000 7f45fcff9700 1 -- 172.21.15.139:0/650587162 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45e80147e0 con 0x7f46000f7920 2021-11-20T06:07:56.893 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.900+0000 7f4605e87700 1 -- 172.21.15.139:0/650587162 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f460010caf0 con 0x7f46000f7920 2021-11-20T06:07:56.893 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.900+0000 7f4605e87700 1 -- 172.21.15.139:0/650587162 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4600027a00 con 0x7f46000f7920 2021-11-20T06:07:56.893 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.901+0000 7f45fcff9700 1 -- 172.21.15.139:0/650587162 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f45e8009480 con 0x7f46000f7920 2021-11-20T06:07:56.894 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.901+0000 7f45fcff9700 1 --2- 172.21.15.139:0/650587162 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f45ec05c2a0 0x7f45ec05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:56.894 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.901+0000 7f45fcff9700 1 -- 172.21.15.139:0/650587162 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f45e8076af0 con 0x7f46000f7920 2021-11-20T06:07:56.894 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.901+0000 7f45ff7fe700 1 --2- 172.21.15.139:0/650587162 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f45ec05c2a0 0x7f45ec05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:56.894 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.902+0000 7f45ff7fe700 1 --2- 172.21.15.139:0/650587162 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f45ec05c2a0 0x7f45ec05e760 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f45f0004e30 tx=0x7f45f0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:56.895 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:56.905+0000 7f45fcff9700 1 -- 172.21.15.139:0/650587162 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45e80492b0 con 0x7f46000f7920 2021-11-20T06:07:57.078 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:57.092+0000 7f4605e87700 1 -- 172.21.15.139:0/650587162 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46000fee10 con 0x7f46000f7920 2021-11-20T06:07:57.078 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:57.092+0000 7f45fcff9700 1 -- 172.21.15.139:0/650587162 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f45e804c8d0 con 0x7f46000f7920 2021-11-20T06:07:57.079 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:57.079 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":81,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:48.265866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24513},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24513":{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":79,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:57.080 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:57.094+0000 7f4605e87700 1 -- 172.21.15.139:0/650587162 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f45ec05c2a0 msgr2=0x7f45ec05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:57.081 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:57.094+0000 7f4605e87700 1 --2- 172.21.15.139:0/650587162 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f45ec05c2a0 0x7f45ec05e760 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f45f0004e30 tx=0x7f45f0009b30).stop 2021-11-20T06:07:57.081 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:57.094+0000 7f4605e87700 1 -- 172.21.15.139:0/650587162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46000f7920 msgr2=0x7f46001073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:57.081 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:57.094+0000 7f4605e87700 1 --2- 172.21.15.139:0/650587162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46000f7920 0x7f46001073a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f45e80026d0 tx=0x7f45e8009de0).stop 2021-11-20T06:07:57.081 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:57.094+0000 7f4605e87700 1 -- 172.21.15.139:0/650587162 shutdown_connections 2021-11-20T06:07:57.082 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:57.094+0000 7f4605e87700 1 --2- 172.21.15.139:0/650587162 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f45ec05c2a0 0x7f45ec05e760 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:57.082 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:57.094+0000 7f4605e87700 1 --2- 172.21.15.139:0/650587162 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46000f6680 0x7f4600106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:57.082 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:57.094+0000 7f4605e87700 1 --2- 172.21.15.139:0/650587162 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46000f7920 0x7f46001073a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:57.082 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:57.094+0000 7f4605e87700 1 -- 172.21.15.139:0/650587162 >> 172.21.15.139:0/650587162 conn(0x7f46000f1de0 msgr2=0x7f46000fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:57.082 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:57.094+0000 7f4605e87700 1 -- 172.21.15.139:0/650587162 shutdown_connections 2021-11-20T06:07:57.083 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:57.094+0000 7f4605e87700 1 -- 172.21.15.139:0/650587162 wait complete. 2021-11-20T06:07:57.083 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 82 2021-11-20T06:07:57.092 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:48.265866+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24513}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24513': {'gid': 24513, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/1490972326', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 1490972326}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 1490972326}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:57.092 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24513:up:rejoin 2021-11-20T06:07:57.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:57 smithi139 conmon[33501]: cluster 2021-11-20T06:07:56.278198+0000 mgr.smithi139.dplyxn (mgr.14162) 324 2021-11-20T06:07:57.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:57 smithi139 conmon[33501]: : cluster [DBG] pgmap v300: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:07:57.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:57 smithi139 conmon[33501]: audit 2021-11-20T06:07:57.094362+0000 mon.smithi139 (mon.0) 857 : audit [DBG] from='client.? 172.21.15.139:0/650587162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:57.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:57 smithi178 conmon[36600]: cluster 2021-11-20T06:07:56.278198+0000 mgr.smithi139.dplyxn (mgr.14162) 324 2021-11-20T06:07:57.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:57 smithi178 conmon[36600]: : cluster [DBG] pgmap v300: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:07:57.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:57 smithi178 conmon[36600]: audit 2021-11-20T06:07:57.094362+0000 mon.smithi139 (mon.0) 857 : audit [DBG] from='client.? 172.21.15.139:0/650587162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:58.094 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:58.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.237+0000 7fa96e706700 1 -- 172.21.15.139:0/1345671654 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa968060490 msgr2=0x7fa9680fb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:58.225 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.237+0000 7fa96e706700 1 --2- 172.21.15.139:0/1345671654 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa968060490 0x7fa9680fb5e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7fa964004660 tx=0x7fa964009b30).stop 2021-11-20T06:07:58.226 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.238+0000 7fa96e706700 1 -- 172.21.15.139:0/1345671654 shutdown_connections 2021-11-20T06:07:58.226 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.238+0000 7fa96e706700 1 --2- 172.21.15.139:0/1345671654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9680fbad0 0x7fa9680fdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:58.226 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.238+0000 7fa96e706700 1 --2- 172.21.15.139:0/1345671654 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa968060490 0x7fa9680fb5e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:58.227 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.238+0000 7fa96e706700 1 -- 172.21.15.139:0/1345671654 >> 172.21.15.139:0/1345671654 conn(0x7fa9680f0c40 msgr2=0x7fa9680f3080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:58.227 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.238+0000 7fa96e706700 1 -- 172.21.15.139:0/1345671654 shutdown_connections 2021-11-20T06:07:58.227 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.238+0000 7fa96e706700 1 -- 172.21.15.139:0/1345671654 wait complete. 2021-11-20T06:07:58.228 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.238+0000 7fa96e706700 1 Processor -- start 2021-11-20T06:07:58.228 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.238+0000 7fa96e706700 1 -- start start 2021-11-20T06:07:58.228 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.238+0000 7fa96e706700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa968060490 0x7fa968108f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:58.228 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.238+0000 7fa96e706700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9680fbad0 0x7fa968109410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:58.229 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.238+0000 7fa96e706700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9681099c0 con 0x7fa9680fbad0 2021-11-20T06:07:58.229 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.238+0000 7fa96e706700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa968109b00 con 0x7fa968060490 2021-11-20T06:07:58.229 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.238+0000 7fa96cf03700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9680fbad0 0x7fa968109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:58.229 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.238+0000 7fa96cf03700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9680fbad0 0x7fa968109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59030/0 (socket says 172.21.15.139:59030) 2021-11-20T06:07:58.229 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.239+0000 7fa96cf03700 1 -- 172.21.15.139:0/4056479307 learned_addr learned my addr 172.21.15.139:0/4056479307 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:58.230 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.239+0000 7fa96d704700 1 --2- 172.21.15.139:0/4056479307 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa968060490 0x7fa968108f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:58.230 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.239+0000 7fa96cf03700 1 -- 172.21.15.139:0/4056479307 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa968060490 msgr2=0x7fa968108f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:58.230 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.239+0000 7fa96cf03700 1 --2- 172.21.15.139:0/4056479307 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa968060490 0x7fa968108f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:58.230 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.239+0000 7fa96cf03700 1 -- 172.21.15.139:0/4056479307 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa964005040 con 0x7fa9680fbad0 2021-11-20T06:07:58.231 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.239+0000 7fa96cf03700 1 --2- 172.21.15.139:0/4056479307 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9680fbad0 0x7fa968109410 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7fa958002700 tx=0x7fa958009d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:58.231 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.239+0000 7fa95dffb700 1 -- 172.21.15.139:0/4056479307 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9580079c0 con 0x7fa9680fbad0 2021-11-20T06:07:58.231 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.239+0000 7fa95dffb700 1 -- 172.21.15.139:0/4056479307 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa95800f040 con 0x7fa9680fbad0 2021-11-20T06:07:58.231 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.239+0000 7fa96e706700 1 -- 172.21.15.139:0/4056479307 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa96810a4a0 con 0x7fa9680fbad0 2021-11-20T06:07:58.232 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.239+0000 7fa95dffb700 1 -- 172.21.15.139:0/4056479307 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa958014740 con 0x7fa9680fbad0 2021-11-20T06:07:58.232 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.239+0000 7fa96e706700 1 -- 172.21.15.139:0/4056479307 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa96810aa60 con 0x7fa9680fbad0 2021-11-20T06:07:58.233 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.240+0000 7fa96e706700 1 -- 172.21.15.139:0/4056479307 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa968045ad0 con 0x7fa9680fbad0 2021-11-20T06:07:58.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.240+0000 7fa95dffb700 1 -- 172.21.15.139:0/4056479307 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa95801b060 con 0x7fa9680fbad0 2021-11-20T06:07:58.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.241+0000 7fa95dffb700 1 --2- 172.21.15.139:0/4056479307 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa95405c300 0x7fa95405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:58.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.241+0000 7fa95dffb700 1 -- 172.21.15.139:0/4056479307 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fa958076230 con 0x7fa9680fbad0 2021-11-20T06:07:58.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.241+0000 7fa96d704700 1 --2- 172.21.15.139:0/4056479307 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa95405c300 0x7fa95405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:58.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.241+0000 7fa96d704700 1 --2- 172.21.15.139:0/4056479307 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa95405c300 0x7fa95405e7c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fa964009f50 tx=0x7fa964009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:58.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.245+0000 7fa95dffb700 1 -- 172.21.15.139:0/4056479307 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9580489f0 con 0x7fa9680fbad0 2021-11-20T06:07:58.414 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.429+0000 7fa96e706700 1 -- 172.21.15.139:0/4056479307 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa968027a00 con 0x7fa9680fbad0 2021-11-20T06:07:58.415 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.429+0000 7fa95dffb700 1 -- 172.21.15.139:0/4056479307 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2187 (secure 0 0 0) 0x7fa95804c010 con 0x7fa9680fbad0 2021-11-20T06:07:58.415 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:58.415 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":81,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:48.265866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24513},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24513":{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":79,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:58.417 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.431+0000 7fa96e706700 1 -- 172.21.15.139:0/4056479307 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa95405c300 msgr2=0x7fa95405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:58.417 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.431+0000 7fa96e706700 1 --2- 172.21.15.139:0/4056479307 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa95405c300 0x7fa95405e7c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fa964009f50 tx=0x7fa964009b30).stop 2021-11-20T06:07:58.417 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.431+0000 7fa96e706700 1 -- 172.21.15.139:0/4056479307 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9680fbad0 msgr2=0x7fa968109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:58.418 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.431+0000 7fa96e706700 1 --2- 172.21.15.139:0/4056479307 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9680fbad0 0x7fa968109410 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7fa958002700 tx=0x7fa958009d60).stop 2021-11-20T06:07:58.418 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.431+0000 7fa96e706700 1 -- 172.21.15.139:0/4056479307 shutdown_connections 2021-11-20T06:07:58.418 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.431+0000 7fa96e706700 1 --2- 172.21.15.139:0/4056479307 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa95405c300 0x7fa95405e7c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:58.418 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.431+0000 7fa96e706700 1 --2- 172.21.15.139:0/4056479307 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa968060490 0x7fa968108f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:58.419 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.431+0000 7fa96e706700 1 --2- 172.21.15.139:0/4056479307 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa9680fbad0 0x7fa968109410 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:58.419 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.431+0000 7fa96e706700 1 -- 172.21.15.139:0/4056479307 >> 172.21.15.139:0/4056479307 conn(0x7fa9680f0c40 msgr2=0x7fa9680f3080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:58.419 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.431+0000 7fa96e706700 1 -- 172.21.15.139:0/4056479307 shutdown_connections 2021-11-20T06:07:58.419 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:58.431+0000 7fa96e706700 1 -- 172.21.15.139:0/4056479307 wait complete. 2021-11-20T06:07:58.419 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 82 2021-11-20T06:07:58.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:48.265866+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24513}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24513': {'gid': 24513, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/1490972326', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 1490972326}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 1490972326}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:58.428 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24513:up:rejoin 2021-11-20T06:07:59.430 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:07:59.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.571+0000 7f3d45a37700 1 -- 172.21.15.139:0/752082977 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d400f4840 msgr2=0x7f3d400f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:59.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.571+0000 7f3d45a37700 1 --2- 172.21.15.139:0/752082977 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d400f4840 0x7f3d400f4c60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f3d34004660 tx=0x7f3d34009b30).stop 2021-11-20T06:07:59.559 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.571+0000 7f3d45a37700 1 -- 172.21.15.139:0/752082977 shutdown_connections 2021-11-20T06:07:59.560 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.571+0000 7f3d45a37700 1 --2- 172.21.15.139:0/752082977 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d400f51e0 0x7f3d400f7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:59.560 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.571+0000 7f3d45a37700 1 --2- 172.21.15.139:0/752082977 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d400f4840 0x7f3d400f4c60 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:59.560 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.571+0000 7f3d45a37700 1 -- 172.21.15.139:0/752082977 >> 172.21.15.139:0/752082977 conn(0x7f3d400f0360 msgr2=0x7f3d400f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:59.560 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.572+0000 7f3d45a37700 1 -- 172.21.15.139:0/752082977 shutdown_connections 2021-11-20T06:07:59.561 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.572+0000 7f3d45a37700 1 -- 172.21.15.139:0/752082977 wait complete. 2021-11-20T06:07:59.561 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.572+0000 7f3d45a37700 1 Processor -- start 2021-11-20T06:07:59.561 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.572+0000 7f3d45a37700 1 -- start start 2021-11-20T06:07:59.561 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.572+0000 7f3d45a37700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d400f4840 0x7f3d40106e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:59.561 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.572+0000 7f3d45a37700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d400f51e0 0x7f3d401072f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:59.562 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.572+0000 7f3d45a37700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d401078f0 con 0x7f3d400f51e0 2021-11-20T06:07:59.562 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.572+0000 7f3d45a37700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d40108100 con 0x7f3d400f4840 2021-11-20T06:07:59.562 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.572+0000 7f3d3ffff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d400f51e0 0x7f3d401072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:59.562 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.573+0000 7f3d3ffff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d400f51e0 0x7f3d401072f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59040/0 (socket says 172.21.15.139:59040) 2021-11-20T06:07:59.563 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.573+0000 7f3d3ffff700 1 -- 172.21.15.139:0/2905624552 learned_addr learned my addr 172.21.15.139:0/2905624552 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:07:59.563 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.573+0000 7f3d44a35700 1 --2- 172.21.15.139:0/2905624552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d400f4840 0x7f3d40106e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:59.563 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.573+0000 7f3d3ffff700 1 -- 172.21.15.139:0/2905624552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d400f4840 msgr2=0x7f3d40106e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:59.563 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.573+0000 7f3d3ffff700 1 --2- 172.21.15.139:0/2905624552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d400f4840 0x7f3d40106e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:59.564 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.573+0000 7f3d3ffff700 1 -- 172.21.15.139:0/2905624552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d34005040 con 0x7f3d400f51e0 2021-11-20T06:07:59.564 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.573+0000 7f3d3ffff700 1 --2- 172.21.15.139:0/2905624552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d400f51e0 0x7f3d401072f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f3d2c0026d0 tx=0x7f3d2c009dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:59.564 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.573+0000 7f3d3d7fa700 1 -- 172.21.15.139:0/2905624552 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d2c0079c0 con 0x7f3d400f51e0 2021-11-20T06:07:59.564 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.573+0000 7f3d45a37700 1 -- 172.21.15.139:0/2905624552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d40108380 con 0x7f3d400f51e0 2021-11-20T06:07:59.564 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.573+0000 7f3d3d7fa700 1 -- 172.21.15.139:0/2905624552 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d2c00f040 con 0x7f3d400f51e0 2021-11-20T06:07:59.565 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.573+0000 7f3d3d7fa700 1 -- 172.21.15.139:0/2905624552 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d2c014840 con 0x7f3d400f51e0 2021-11-20T06:07:59.565 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.573+0000 7f3d45a37700 1 -- 172.21.15.139:0/2905624552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d40108940 con 0x7f3d400f51e0 2021-11-20T06:07:59.566 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.574+0000 7f3d45a37700 1 -- 172.21.15.139:0/2905624552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d400f3ff0 con 0x7f3d400f51e0 2021-11-20T06:07:59.566 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.574+0000 7f3d3d7fa700 1 -- 172.21.15.139:0/2905624552 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3d2c01b060 con 0x7f3d400f51e0 2021-11-20T06:07:59.567 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.575+0000 7f3d3d7fa700 1 --2- 172.21.15.139:0/2905624552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3d3005c300 0x7f3d3005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:07:59.567 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.575+0000 7f3d3d7fa700 1 -- 172.21.15.139:0/2905624552 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f3d2c0762d0 con 0x7f3d400f51e0 2021-11-20T06:07:59.567 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.575+0000 7f3d44a35700 1 --2- 172.21.15.139:0/2905624552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3d3005c300 0x7f3d3005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:07:59.567 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.575+0000 7f3d44a35700 1 --2- 172.21.15.139:0/2905624552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3d3005c300 0x7f3d3005e7c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f3d40106b70 tx=0x7f3d34009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:07:59.568 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.579+0000 7f3d3d7fa700 1 -- 172.21.15.139:0/2905624552 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d2c048a30 con 0x7f3d400f51e0 2021-11-20T06:07:59.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:59 smithi178 conmon[36600]: cluster 2021-11-20T06:07:58.278598+0000 mgr.smithi139.dplyxn ( 2021-11-20T06:07:59.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:59 smithi178 conmon[36600]: mgr.14162) 325 : cluster [DBG] pgmap v301: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:07:59.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:07:59 smithi178 conmon[36600]: audit 2021-11-20T06:07:58.430918+0000 mon.smithi139 (mon.0) 858 : audit [DBG] from='client.? 172.21.15.139:0/4056479307' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:59.748 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.763+0000 7f3d45a37700 1 -- 172.21.15.139:0/2905624552 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d400feee0 con 0x7f3d400f51e0 2021-11-20T06:07:59.749 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.763+0000 7f3d3d7fa700 1 -- 172.21.15.139:0/2905624552 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f3d400feee0 con 0x7f3d400f51e0 2021-11-20T06:07:59.749 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:07:59.749 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":81,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:07:48.265866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24513},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24513":{"gid":24513,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":79,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/1490972326","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1490972326},{"type":"v1","addr":"172.21.15.178:6825","nonce":1490972326}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:07:59.751 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.765+0000 7f3d45a37700 1 -- 172.21.15.139:0/2905624552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3d3005c300 msgr2=0x7f3d3005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:59.751 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.765+0000 7f3d45a37700 1 --2- 172.21.15.139:0/2905624552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3d3005c300 0x7f3d3005e7c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f3d40106b70 tx=0x7f3d34009b30).stop 2021-11-20T06:07:59.751 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.765+0000 7f3d45a37700 1 -- 172.21.15.139:0/2905624552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d400f51e0 msgr2=0x7f3d401072f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:07:59.752 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.765+0000 7f3d45a37700 1 --2- 172.21.15.139:0/2905624552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d400f51e0 0x7f3d401072f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f3d2c0026d0 tx=0x7f3d2c009dc0).stop 2021-11-20T06:07:59.752 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.765+0000 7f3d45a37700 1 -- 172.21.15.139:0/2905624552 shutdown_connections 2021-11-20T06:07:59.752 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.765+0000 7f3d45a37700 1 --2- 172.21.15.139:0/2905624552 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3d3005c300 0x7f3d3005e7c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:59.752 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.765+0000 7f3d45a37700 1 --2- 172.21.15.139:0/2905624552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d400f4840 0x7f3d40106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:59.753 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.765+0000 7f3d45a37700 1 --2- 172.21.15.139:0/2905624552 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3d400f51e0 0x7f3d401072f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:07:59.753 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.765+0000 7f3d45a37700 1 -- 172.21.15.139:0/2905624552 >> 172.21.15.139:0/2905624552 conn(0x7f3d400f0360 msgr2=0x7f3d400fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:07:59.753 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.765+0000 7f3d45a37700 1 -- 172.21.15.139:0/2905624552 shutdown_connections 2021-11-20T06:07:59.753 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:07:59.765+0000 7f3d45a37700 1 -- 172.21.15.139:0/2905624552 wait complete. 2021-11-20T06:07:59.754 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 82 2021-11-20T06:07:59.762 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:59 smithi139 conmon[33501]: cluster 2021-11-20T06:07:58.278598+0000 mgr.smithi139.dplyxn (mgr.14162) 325 : cluster [DBG] pgmap v301: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:07:59.762 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:07:59 smithi139 conmon[33501]: audit 2021-11-20T06:07:58.430918+0000 mon.smithi139 (mon.0) 858 : audit [DBG] from='client.? 172.21.15.139:0/4056479307' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:07:59.763 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:07:48.265866+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24513}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24513': {'gid': 24513, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/1490972326', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 1490972326}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 1490972326}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:07:59.764 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24513:up:rejoin 2021-11-20T06:08:00.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:00 smithi178 conmon[36600]: cluster 2021-11-20T06:07:59.483296+0000 mon.smithi139 (mon.0) 859 : cluster [INF] 2021-11-20T06:08:00.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:00 smithi178 conmon[36600]: daemon mds.cephfs.smithi178.blvkbx restarted 2021-11-20T06:08:00.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:00 smithi178 conmon[36600]: cluster 2021-11-20T06:07:59.491771+0000 mon.smithi139 (mon.0) 860 : cluster [DBG] osdmap e57: 6 total, 6 up, 6 in 2021-11-20T06:08:00.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:00 smithi178 conmon[36600]: audit 2021-11-20T06:07:59 2021-11-20T06:08:00.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:00 smithi178 conmon[36600]: .764943+0000 mon.smithi139 (mon.0) 861 : audit [DBG] from='client.? 172.21.15.139:0/2905624552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:00.765 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:00.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:00 smithi139 conmon[33501]: debug 2021-11-20T06:08:00.302+0000 7f2a21654700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:08:00.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:00 smithi139 conmon[33501]: cluster 2021-11-20T06:07:59.483296+0000 mon.smithi139 (mon.0) 859 : cluster [INF] daemon mds.cephfs.smithi178.blvkbx restarted 2021-11-20T06:08:00.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:00 smithi139 conmon[33501]: cluster 2021-11-20T06:07:59.491771+0000 mon.smithi139 (mon.0) 860 : cluster [DBG] osdmap e57: 6 total, 6 up, 6 in 2021-11-20T06:08:00.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:00 smithi139 conmon[33501]: audit 2021-11-20T06:07:59.764943+0000 mon.smithi139 (mon 2021-11-20T06:08:00.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:00 smithi139 conmon[33501]: .0) 861 : audit [DBG] from='client.? 172.21.15.139:0/2905624552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:00.895 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.907+0000 7ffbbbca8700 1 -- 172.21.15.139:0/2480464943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbb40f55a0 msgr2=0x7ffbb40f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:00.895 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.907+0000 7ffbbbca8700 1 --2- 172.21.15.139:0/2480464943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbb40f55a0 0x7ffbb40f59c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7ffba40099e0 tx=0x7ffba4004d10).stop 2021-11-20T06:08:00.896 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.907+0000 7ffbbbca8700 1 -- 172.21.15.139:0/2480464943 shutdown_connections 2021-11-20T06:08:00.896 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.907+0000 7ffbbbca8700 1 --2- 172.21.15.139:0/2480464943 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffbb40f6840 0x7ffbb40f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:08:00.896 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.907+0000 7ffbbbca8700 1 --2- 172.21.15.139:0/2480464943 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbb40f55a0 0x7ffbb40f59c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:00.896 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.907+0000 7ffbbbca8700 1 -- 172.21.15.139:0/2480464943 >> 172.21.15.139:0/2480464943 conn(0x7ffbb40f0ce0 msgr2=0x7ffbb40f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:00.897 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.907+0000 7ffbbbca8700 1 -- 172.21.15.139:0/2480464943 shutdown_connections 2021-11-20T06:08:00.897 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.907+0000 7ffbbbca8700 1 -- 172.21.15.139:0/2480464943 wait complete. 2021-11-20T06:08:00.897 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.907+0000 7ffbbbca8700 1 Processor -- start 2021-11-20T06:08:00.897 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.907+0000 7ffbbbca8700 1 -- start start 2021-11-20T06:08:00.898 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbbbca8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbb40f55a0 0x7ffbb405ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:00.898 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbbbca8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffbb40f6840 0x7ffbb410be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:00.898 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbbbca8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb410c310 con 0x7ffbb40f55a0 2021-11-20T06:08:00.898 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbbbca8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb410c450 con 0x7ffbb40f6840 2021-11-20T06:08:00.899 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbb9a44700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbb40f55a0 0x7ffbb405ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:00.899 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbb9a44700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbb40f55a0 0x7ffbb405ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59048/0 (socket says 172.21.15.139:59048) 2021-11-20T06:08:00.899 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbb9a44700 1 -- 172.21.15.139:0/3396335768 learned_addr learned my addr 172.21.15.139:0/3396335768 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:00.899 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbb9243700 1 --2- 172.21.15.139:0/3396335768 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffbb40f6840 0x7ffbb410be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:00.899 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbb9a44700 1 -- 172.21.15.139:0/3396335768 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffbb40f6840 msgr2=0x7ffbb410be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:00.900 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbb9a44700 1 --2- 172.21.15.139:0/3396335768 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffbb40f6840 0x7ffbb410be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:00.900 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbb9a44700 1 -- 172.21.15.139:0/3396335768 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba4009640 con 0x7ffbb40f55a0 2021-11-20T06:08:00.900 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbb9a44700 1 --2- 172.21.15.139:0/3396335768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbb40f55a0 0x7ffbb405ff20 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7ffba4007830 tx=0x7ffba4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:00.900 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbb27fc700 1 -- 172.21.15.139:0/3396335768 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffba400e030 con 0x7ffbb40f55a0 2021-11-20T06:08:00.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbb27fc700 1 -- 172.21.15.139:0/3396335768 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffba401d8d0 con 0x7ffbb40f55a0 2021-11-20T06:08:00.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbbbca8700 1 -- 172.21.15.139:0/3396335768 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb410c6d0 con 0x7ffbb40f55a0 2021-11-20T06:08:00.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbb27fc700 1 -- 172.21.15.139:0/3396335768 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffba4018070 con 0x7ffbb40f55a0 2021-11-20T06:08:00.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.908+0000 7ffbbbca8700 1 -- 172.21.15.139:0/3396335768 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb410cbb0 con 0x7ffbb40f55a0 2021-11-20T06:08:00.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.909+0000 7ffbbbca8700 1 -- 172.21.15.139:0/3396335768 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb4027a00 con 0x7ffbb40f55a0 2021-11-20T06:08:00.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.909+0000 7ffbb27fc700 1 -- 172.21.15.139:0/3396335768 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ffba401da40 con 0x7ffbb40f55a0 2021-11-20T06:08:00.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.910+0000 7ffbb27fc700 1 --2- 172.21.15.139:0/3396335768 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ffb9c05c2a0 0x7ffb9c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:00.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.910+0000 7ffbb27fc700 1 -- 172.21.15.139:0/3396335768 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7ffba4078330 con 0x7ffbb40f55a0 2021-11-20T06:08:00.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.910+0000 7ffbb9243700 1 --2- 172.21.15.139:0/3396335768 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ffb9c05c2a0 0x7ffb9c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:00.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.910+0000 7ffbb9243700 1 --2- 172.21.15.139:0/3396335768 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ffb9c05c2a0 0x7ffb9c05e760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7ffba800b790 tx=0x7ffba8008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:00.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:00.914+0000 7ffbb27fc700 1 -- 172.21.15.139:0/3396335768 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffba404aa90 con 0x7ffbb40f55a0 2021-11-20T06:08:01.082 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:01.096+0000 7ffbbbca8700 1 -- 172.21.15.139:0/3396335768 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbb40fddb0 con 0x7ffbb40f55a0 2021-11-20T06:08:01.083 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:01.097+0000 7ffbb27fc700 1 -- 172.21.15.139:0/3396335768 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+2188 (secure 0 0 0) 0x7ffba404e0b0 con 0x7ffbb40f55a0 2021-11-20T06:08:01.083 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:01.083 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":84,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:00.309334+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24549},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24549":{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":84,"state":"up:replay","state_seq":3,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:01.085 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:01.099+0000 7ffbbbca8700 1 -- 172.21.15.139:0/3396335768 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ffb9c05c2a0 msgr2=0x7ffb9c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:01.085 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:01.099+0000 7ffbbbca8700 1 --2- 172.21.15.139:0/3396335768 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ffb9c05c2a0 0x7ffb9c05e760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7ffba800b790 tx=0x7ffba8008040).stop 2021-11-20T06:08:01.085 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:01.099+0000 7ffbbbca8700 1 -- 172.21.15.139:0/3396335768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbb40f55a0 msgr2=0x7ffbb405ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:01.086 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:01.099+0000 7ffbbbca8700 1 --2- 172.21.15.139:0/3396335768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbb40f55a0 0x7ffbb405ff20 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7ffba4007830 tx=0x7ffba4004d10).stop 2021-11-20T06:08:01.086 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:01.099+0000 7ffbbbca8700 1 -- 172.21.15.139:0/3396335768 shutdown_connections 2021-11-20T06:08:01.086 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:01.099+0000 7ffbbbca8700 1 --2- 172.21.15.139:0/3396335768 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ffb9c05c2a0 0x7ffb9c05e760 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:01.086 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:01.099+0000 7ffbbbca8700 1 --2- 172.21.15.139:0/3396335768 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ffbb40f55a0 0x7ffbb405ff20 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:01.087 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:01.099+0000 7ffbbbca8700 1 --2- 172.21.15.139:0/3396335768 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffbb40f6840 0x7ffbb410be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:01.088 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:01.099+0000 7ffbbbca8700 1 -- 172.21.15.139:0/3396335768 >> 172.21.15.139:0/3396335768 conn(0x7ffbb40f0ce0 msgr2=0x7ffbb40fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:01.088 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:01.099+0000 7ffbbbca8700 1 -- 172.21.15.139:0/3396335768 shutdown_connections 2021-11-20T06:08:01.089 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:01.099+0000 7ffbbbca8700 1 -- 172.21.15.139:0/3396335768 wait complete. 2021-11-20T06:08:01.089 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 84 2021-11-20T06:08:01.096 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:00.309334+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24549}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24549': {'gid': 24549, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 84, 'state': 'up:replay', 'state_seq': 3, 'addr': '172.21.15.178:6827/2304740608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2304740608}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2304740608}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:01.097 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24549:up:replay 2021-11-20T06:08:01.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: cluster 2021-11-20T06:08:00.278884+0000 mgr.smithi139.dplyxn (mgr.14162) 326 : cluster [DBG] pgmap v303: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2021-11-20T06:08:01.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: cluster 2021-11-20T 2021-11-20T06:08:01.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: 06:08:00.303724+0000 mon.smithi139 (mon.0) 862 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:08:01.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: cluster 2021-11-20T06:08:00.309234+0000 mon.smithi139 (mon.0) 863 : cluster [DBG] mds.? [v2:172.21.15.178:6824/4081910640,v1:172.21.15.178:6825/4081910640] up:boot 2021-11-20T06:08:01.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: cluster 2021-11-20T06:08:00.309298+0000 2021-11-20T06:08:01.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: mon.smithi139 (mon.0) 864 : cluster [INF] Standby daemon mds.cephfs.smithi178.pskihy assigned to filesystem cephfs as rank 0 2021-11-20T06:08:01.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: cluster 2021 2021-11-20T06:08:01.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: -11-20T06:08:00.309393+0000 2021-11-20T06:08:01.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: mon.smithi139 (mon.0) 865 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:08:01.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: 2021-11-20T06:08:01.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: cluster 2021-11-20T06:08:01.529 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: 2021-11-20T06:08:00.309604+0000 mon.smithi139 (mon 2021-11-20T06:08:01.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: .0) 866 : cluster [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T06:08:01.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: audit 2021-11- 2021-11-20T06:08:01.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: 20T06:08:00.309715+0000 mon.smithi139 ( 2021-11-20T06:08:01.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: mon.0) 867 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.blvkbx"}]: dispatch 2021-11-20T06:08:01.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: cluster 2021 2021-11-20T06:08:01.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: -11-20T06:08:00.312066+0000 2021-11-20T06:08:01.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: mon.smithi139 (mon.0) 868 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:replay} 1 up:standby 2021-11-20T06:08:01.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: 2021-11-20T06:08:01.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: audit 2021-11-20T06:08:01.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: 2021-11-20T06:08:01.098736+0000 mon.smithi139 ( 2021-11-20T06:08:01.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:01 smithi139 conmon[33501]: mon.0) 869 : audit [DBG] from='client.? 172.21.15.139:0/3396335768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:01.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: cluster 2021-11-20T06:08:00.278884+0000 mgr.smithi139.dplyxn (mgr.14162) 326 : cluster [DBG] pgmap v303: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2021-11-20T06:08:01.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: cluster 2021-11-20T06:08:00.303724+0000 mon.smithi139 (mon.0) 862 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:08:01.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: cluster 2021-11-20T06:08:00.309234+0000 mon.smithi139 (mon.0) 863 : cluster [DBG] 2021-11-20T06:08:01.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: mds.? [v2:172.21.15.178:6824/4081910640,v1:172.21.15.178:6825/4081910640] up:boot 2021-11-20T06:08:01.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: cluster 2021-11-20T06:08:00.309298+0000 2021-11-20T06:08:01.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: mon.smithi139 (mon.0) 864 : cluster [INF] Standby daemon mds.cephfs.smithi178.pskihy assigned to filesystem cephfs as rank 0 2021-11-20T06:08:01.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: cluster 2021-11-20T 2021-11-20T06:08:01.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: 06:08:00.309393+0000 mon.smithi139 (mon. 2021-11-20T06:08:01.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: 0) 865 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-20T06:08:01.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: cluster 2021-11 2021-11-20T06:08:01.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: -20T06:08:00.309604+0000 mon.smithi139 ( 2021-11-20T06:08:01.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: mon.0) 866 : cluster [DBG] fsmap cephfs:0/1 2 up:standby, 1 failed 2021-11-20T06:08:01.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: audit 2021-11- 2021-11-20T06:08:01.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: 20T06:08:00.309715+0000 mon.smithi139 (mon 2021-11-20T06:08:01.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: .0) 867 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.blvkbx"}]: dispatch 2021-11-20T06:08:01.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: cluster 2021-11- 2021-11-20T06:08:01.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: 20T06:08:00.312066+0000 mon.smithi139 ( 2021-11-20T06:08:01.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: mon.0) 868 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:replay} 1 up:standby 2021-11-20T06:08:01.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: audit 2021-11 2021-11-20T06:08:01.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: -20T06:08:01.098736+0000 mon.smithi139 2021-11-20T06:08:01.696 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:01 smithi178 conmon[36600]: (mon.0) 869 : audit [DBG] from='client.? 172.21.15.139:0/3396335768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:02.098 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:02.232 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.245+0000 7ff0510b1700 1 -- 172.21.15.139:0/1077910335 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff04c0f55c0 msgr2=0x7ff04c0f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:02.233 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.245+0000 7ff0510b1700 1 --2- 172.21.15.139:0/1077910335 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff04c0f55c0 0x7ff04c0f59e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7ff03c004660 tx=0x7ff03c009b30).stop 2021-11-20T06:08:02.233 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.245+0000 7ff0510b1700 1 -- 172.21.15.139:0/1077910335 shutdown_connections 2021-11-20T06:08:02.233 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.245+0000 7ff0510b1700 1 --2- 172.21.15.139:0/1077910335 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff04c0f6860 0x7ff04c0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:02.233 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.245+0000 7ff0510b1700 1 --2- 172.21.15.139:0/1077910335 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff04c0f55c0 0x7ff04c0f59e0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:02.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.245+0000 7ff0510b1700 1 -- 172.21.15.139:0/1077910335 >> 172.21.15.139:0/1077910335 conn(0x7ff04c0f0d40 msgr2=0x7ff04c0f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:02.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.245+0000 7ff0510b1700 1 -- 172.21.15.139:0/1077910335 shutdown_connections 2021-11-20T06:08:02.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.245+0000 7ff0510b1700 1 -- 172.21.15.139:0/1077910335 wait complete. 2021-11-20T06:08:02.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.246+0000 7ff0510b1700 1 Processor -- start 2021-11-20T06:08:02.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.246+0000 7ff0510b1700 1 -- start start 2021-11-20T06:08:02.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.246+0000 7ff0510b1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff04c0f55c0 0x7ff04c0604d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:02.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.246+0000 7ff0510b1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff04c0f6860 0x7ff04c10bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:02.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.246+0000 7ff0510b1700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c10c180 con 0x7ff04c0f6860 2021-11-20T06:08:02.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.246+0000 7ff0510b1700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c10c2c0 con 0x7ff04c0f55c0 2021-11-20T06:08:02.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.246+0000 7ff043fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff04c0f6860 0x7ff04c10bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:02.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.246+0000 7ff043fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff04c0f6860 0x7ff04c10bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59060/0 (socket says 172.21.15.139:59060) 2021-11-20T06:08:02.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.246+0000 7ff043fff700 1 -- 172.21.15.139:0/1345624413 learned_addr learned my addr 172.21.15.139:0/1345624413 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:02.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.246+0000 7ff04ad9d700 1 --2- 172.21.15.139:0/1345624413 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff04c0f55c0 0x7ff04c0604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:02.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.247+0000 7ff043fff700 1 -- 172.21.15.139:0/1345624413 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff04c0f55c0 msgr2=0x7ff04c0604d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:02.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.247+0000 7ff043fff700 1 --2- 172.21.15.139:0/1345624413 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff04c0f55c0 0x7ff04c0604d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:02.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.247+0000 7ff043fff700 1 -- 172.21.15.139:0/1345624413 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03c005040 con 0x7ff04c0f6860 2021-11-20T06:08:02.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.247+0000 7ff043fff700 1 --2- 172.21.15.139:0/1345624413 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff04c0f6860 0x7ff04c10bc90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7ff03400e7f0 tx=0x7ff03400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:02.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.248+0000 7ff0437fe700 1 -- 172.21.15.139:0/1345624413 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0340092e0 con 0x7ff04c0f6860 2021-11-20T06:08:02.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.248+0000 7ff0437fe700 1 -- 172.21.15.139:0/1345624413 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff03400f040 con 0x7ff04c0f6860 2021-11-20T06:08:02.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.248+0000 7ff0510b1700 1 -- 172.21.15.139:0/1345624413 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff04c10c520 con 0x7ff04c0f6860 2021-11-20T06:08:02.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.248+0000 7ff0437fe700 1 -- 172.21.15.139:0/1345624413 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff034014750 con 0x7ff04c0f6860 2021-11-20T06:08:02.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.248+0000 7ff0510b1700 1 -- 172.21.15.139:0/1345624413 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04c10cab0 con 0x7ff04c0f6860 2021-11-20T06:08:02.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.249+0000 7ff0510b1700 1 -- 172.21.15.139:0/1345624413 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04c046330 con 0x7ff04c0f6860 2021-11-20T06:08:02.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.250+0000 7ff0437fe700 1 -- 172.21.15.139:0/1345624413 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff03401b030 con 0x7ff04c0f6860 2021-11-20T06:08:02.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.250+0000 7ff0437fe700 1 --2- 172.21.15.139:0/1345624413 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff03805c2a0 0x7ff03805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:02.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.250+0000 7ff0437fe700 1 -- 172.21.15.139:0/1345624413 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7ff034076130 con 0x7ff04c0f6860 2021-11-20T06:08:02.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.250+0000 7ff04ad9d700 1 --2- 172.21.15.139:0/1345624413 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff03805c2a0 0x7ff03805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:02.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.250+0000 7ff04ad9d700 1 --2- 172.21.15.139:0/1345624413 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff03805c2a0 0x7ff03805e760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7ff03c013530 tx=0x7ff03c016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:02.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.254+0000 7ff0437fe700 1 -- 172.21.15.139:0/1345624413 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff034048940 con 0x7ff04c0f6860 2021-11-20T06:08:02.424 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.439+0000 7ff0510b1700 1 -- 172.21.15.139:0/1345624413 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff04c027a00 con 0x7ff04c0f6860 2021-11-20T06:08:02.425 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.439+0000 7ff0437fe700 1 -- 172.21.15.139:0/1345624413 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2188 (secure 0 0 0) 0x7ff034019090 con 0x7ff04c0f6860 2021-11-20T06:08:02.425 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:02.425 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:02.313577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24549},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24549":{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:02.427 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.441+0000 7ff0510b1700 1 -- 172.21.15.139:0/1345624413 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff03805c2a0 msgr2=0x7ff03805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:02.427 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.441+0000 7ff0510b1700 1 --2- 172.21.15.139:0/1345624413 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff03805c2a0 0x7ff03805e760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7ff03c013530 tx=0x7ff03c016040).stop 2021-11-20T06:08:02.427 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.441+0000 7ff0510b1700 1 -- 172.21.15.139:0/1345624413 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff04c0f6860 msgr2=0x7ff04c10bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:02.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.441+0000 7ff0510b1700 1 --2- 172.21.15.139:0/1345624413 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff04c0f6860 0x7ff04c10bc90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7ff03400e7f0 tx=0x7ff03400b040).stop 2021-11-20T06:08:02.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.441+0000 7ff0510b1700 1 -- 172.21.15.139:0/1345624413 shutdown_connections 2021-11-20T06:08:02.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.441+0000 7ff0510b1700 1 --2- 172.21.15.139:0/1345624413 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7ff03805c2a0 0x7ff03805e760 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:02.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.441+0000 7ff0510b1700 1 --2- 172.21.15.139:0/1345624413 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff04c0f55c0 0x7ff04c0604d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:02.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.441+0000 7ff0510b1700 1 --2- 172.21.15.139:0/1345624413 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff04c0f6860 0x7ff04c10bc90 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:02.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.441+0000 7ff0510b1700 1 -- 172.21.15.139:0/1345624413 >> 172.21.15.139:0/1345624413 conn(0x7ff04c0f0d40 msgr2=0x7ff04c0fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:02.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.441+0000 7ff0510b1700 1 -- 172.21.15.139:0/1345624413 shutdown_connections 2021-11-20T06:08:02.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:02.441+0000 7ff0510b1700 1 -- 172.21.15.139:0/1345624413 wait complete. 2021-11-20T06:08:02.429 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 86 2021-11-20T06:08:02.437 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:02 smithi139 conmon[33501]: cluster 2021-11-20T06:08:01.314880+0000 mon.smithi139 (mon.0) 870 : cluster [DBG] mds.? [v2:172.21.15.178:6826/2304740608,v1:172.21.15.178:6827/2304740608] up:reconnect 2021-11-20T06:08:02.438 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:02 smithi139 conmon[33501]: 2021-11-20T06:08:02.438 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:02 smithi139 conmon[33501]: cluster 2021-11-20T06:08:01.314937+0000 mon.smithi139 (mon.0) 871 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:reconnect} 1 up:standby 2021-11-20T06:08:02.440 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:02.313577+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24549}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24549': {'gid': 24549, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 84, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6827/2304740608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2304740608}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2304740608}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:02.440 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24549:up:rejoin 2021-11-20T06:08:02.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:02 smithi178 conmon[36600]: cluster 2021-11-20T06:08:01.314880+0000 mon.smithi139 (mon.0) 870 2021-11-20T06:08:02.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:02 smithi178 conmon[36600]: : cluster [DBG] mds.? [v2:172.21.15.178:6826/2304740608,v1:172.21.15.178:6827/2304740608] up:reconnect 2021-11-20T06:08:02.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:02 smithi178 conmon[36600]: cluster 2021-11-20T06:08:01.314937+0000 mon.smithi139 (mon.0) 871 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:reconnect} 1 up:standby 2021-11-20T06:08:03.441 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:03.469 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:03 smithi139 conmon[33501]: cluster 2021-11-20T06:08:02.279233+0000 mgr.smithi139.dplyxn (mgr.14162) 327 : cluster [DBG] pgmap v304: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-20T06:08:03.469 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:03 smithi139 conmon[33501]: cluster 2021-11-20T06:08:02.318721+0000 mon.smithi139 (mon.0) 872 : cluster [DBG] mds.? [v2:172.21.15.178:6826/2304740608,v1:172.21.15.178:6827/2304740608] up:rejoin 2021-11-20T06:08:03.469 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:03 smithi139 conmon[33501]: cluster 2021-11-20T06:08:02.318790+0000 mon.smithi139 (mon.0) 873 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:rejoin} 1 up:standby 2021-11-20T06:08:03.470 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:03 smithi139 conmon[33501]: audit 2021-11-20T06:08:02.441000+0000 mon.smithi139 (mon 2021-11-20T06:08:03.470 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:03 smithi139 conmon[33501]: .0) 874 : audit [DBG] from='client.? 172.21.15.139:0/1345624413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:03.575 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.588+0000 7f045b414700 1 -- 172.21.15.139:0/890423794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04540f9810 msgr2=0x7f04540fbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:03.575 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.588+0000 7f045b414700 1 --2- 172.21.15.139:0/890423794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04540f9810 0x7f04540fbc40 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f0444004660 tx=0x7f0444009b30).stop 2021-11-20T06:08:03.576 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.588+0000 7f045b414700 1 -- 172.21.15.139:0/890423794 shutdown_connections 2021-11-20T06:08:03.576 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.588+0000 7f045b414700 1 --2- 172.21.15.139:0/890423794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04540f9810 0x7f04540fbc40 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:03.576 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.588+0000 7f045b414700 1 --2- 172.21.15.139:0/890423794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f04540606d0 0x7f04540f9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:03.576 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.588+0000 7f045b414700 1 -- 172.21.15.139:0/890423794 >> 172.21.15.139:0/890423794 conn(0x7f04540f0ce0 msgr2=0x7f04540f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:03.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.588+0000 7f045b414700 1 -- 172.21.15.139:0/890423794 shutdown_connections 2021-11-20T06:08:03.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.588+0000 7f045b414700 1 -- 172.21.15.139:0/890423794 wait complete. 2021-11-20T06:08:03.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.588+0000 7f045b414700 1 Processor -- start 2021-11-20T06:08:03.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.588+0000 7f045b414700 1 -- start start 2021-11-20T06:08:03.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f045b414700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04540606d0 0x7f0454107020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:03.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f045b414700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0454107510 0x7f045410c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:03.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f045b414700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04541079e0 con 0x7f04540606d0 2021-11-20T06:08:03.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f045b414700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0454107b20 con 0x7f0454107510 2021-11-20T06:08:03.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f04591b0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04540606d0 0x7f0454107020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:03.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f04591b0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04540606d0 0x7f0454107020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59072/0 (socket says 172.21.15.139:59072) 2021-11-20T06:08:03.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f04591b0700 1 -- 172.21.15.139:0/2575779243 learned_addr learned my addr 172.21.15.139:0/2575779243 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:03.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f04589af700 1 --2- 172.21.15.139:0/2575779243 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0454107510 0x7f045410c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:03.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f04591b0700 1 -- 172.21.15.139:0/2575779243 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0454107510 msgr2=0x7f045410c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:03.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f04591b0700 1 --2- 172.21.15.139:0/2575779243 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0454107510 0x7f045410c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:03.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f04591b0700 1 -- 172.21.15.139:0/2575779243 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0444005040 con 0x7f04540606d0 2021-11-20T06:08:03.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f04591b0700 1 --2- 172.21.15.139:0/2575779243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04540606d0 0x7f0454107020 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f04500026d0 tx=0x7f0450009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:03.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f0449ffb700 1 -- 172.21.15.139:0/2575779243 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04500079c0 con 0x7f04540606d0 2021-11-20T06:08:03.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f0449ffb700 1 -- 172.21.15.139:0/2575779243 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f045000f040 con 0x7f04540606d0 2021-11-20T06:08:03.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f045b414700 1 -- 172.21.15.139:0/2575779243 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045410cad0 con 0x7f04540606d0 2021-11-20T06:08:03.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.589+0000 7f0449ffb700 1 -- 172.21.15.139:0/2575779243 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04500148b0 con 0x7f04540606d0 2021-11-20T06:08:03.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.590+0000 7f045b414700 1 -- 172.21.15.139:0/2575779243 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f045410d030 con 0x7f04540606d0 2021-11-20T06:08:03.582 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.591+0000 7f0449ffb700 1 -- 172.21.15.139:0/2575779243 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0450014a10 con 0x7f04540606d0 2021-11-20T06:08:03.582 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.591+0000 7f045b414700 1 -- 172.21.15.139:0/2575779243 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0454027a00 con 0x7f04540606d0 2021-11-20T06:08:03.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.591+0000 7f0449ffb700 1 --2- 172.21.15.139:0/2575779243 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f044005c2a0 0x7f044005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:03.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.591+0000 7f0449ffb700 1 -- 172.21.15.139:0/2575779243 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f0450076390 con 0x7f04540606d0 2021-11-20T06:08:03.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.591+0000 7f04589af700 1 --2- 172.21.15.139:0/2575779243 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f044005c2a0 0x7f044005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:03.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.591+0000 7f04589af700 1 --2- 172.21.15.139:0/2575779243 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f044005c2a0 0x7f044005e760 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f0444008000 tx=0x7f0444009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:03.584 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.595+0000 7f0449ffb700 1 -- 172.21.15.139:0/2575779243 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f045004c520 con 0x7f04540606d0 2021-11-20T06:08:03.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:03 smithi178 conmon[36600]: cluster 2021-11-20T06:08:02.279233+0000 mgr.smithi139.dplyxn (mgr.14162) 327 : cluster [DBG] pgmap v304: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-20T06:08:03.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:03 smithi178 conmon[36600]: cluster 2021-11-20T06:08:02.318721+0000 mon.smithi139 (mon.0) 872 : cluster [DBG] mds.? [v2:172.21.15.178:6826/2304740608,v1:172.21.15.178:6827/2304740608] up:rejoin 2021-11-20T06:08:03.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:03 smithi178 conmon[36600]: cluster 2021-11-20T06:08:02.318790+0000 mon.smithi139 (mon.0) 873 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:rejoin} 1 up:standby 2021-11-20T06:08:03.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:03 smithi178 conmon[36600]: audit 2021-11-20T06:08:02.441000+0000 mon.smithi139 (mon.0) 2021-11-20T06:08:03.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:03 smithi178 conmon[36600]: 874 : audit [DBG] from='client.? 172.21.15.139:0/1345624413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:03.764 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.778+0000 7f045b414700 1 -- 172.21.15.139:0/2575779243 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04540fee20 con 0x7f04540606d0 2021-11-20T06:08:03.765 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.779+0000 7f0449ffb700 1 -- 172.21.15.139:0/2575779243 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f045004c110 con 0x7f04540606d0 2021-11-20T06:08:03.765 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:03.765 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:02.313577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24549},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24549":{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:03.767 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.781+0000 7f045b414700 1 -- 172.21.15.139:0/2575779243 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f044005c2a0 msgr2=0x7f044005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:03.767 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.781+0000 7f045b414700 1 --2- 172.21.15.139:0/2575779243 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f044005c2a0 0x7f044005e760 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f0444008000 tx=0x7f0444009b30).stop 2021-11-20T06:08:03.767 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.781+0000 7f045b414700 1 -- 172.21.15.139:0/2575779243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04540606d0 msgr2=0x7f0454107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:03.767 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.781+0000 7f045b414700 1 --2- 172.21.15.139:0/2575779243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04540606d0 0x7f0454107020 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f04500026d0 tx=0x7f0450009de0).stop 2021-11-20T06:08:03.768 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.781+0000 7f045b414700 1 -- 172.21.15.139:0/2575779243 shutdown_connections 2021-11-20T06:08:03.768 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.781+0000 7f045b414700 1 --2- 172.21.15.139:0/2575779243 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f044005c2a0 0x7f044005e760 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:03.768 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.781+0000 7f045b414700 1 --2- 172.21.15.139:0/2575779243 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f04540606d0 0x7f0454107020 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:03.768 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.781+0000 7f045b414700 1 --2- 172.21.15.139:0/2575779243 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0454107510 0x7f045410c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:03.768 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.781+0000 7f045b414700 1 -- 172.21.15.139:0/2575779243 >> 172.21.15.139:0/2575779243 conn(0x7f04540f0ce0 msgr2=0x7f04540f30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:03.769 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.781+0000 7f045b414700 1 -- 172.21.15.139:0/2575779243 shutdown_connections 2021-11-20T06:08:03.769 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:03.781+0000 7f045b414700 1 -- 172.21.15.139:0/2575779243 wait complete. 2021-11-20T06:08:03.769 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 86 2021-11-20T06:08:03.778 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:02.313577+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24549}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24549': {'gid': 24549, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 84, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6827/2304740608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2304740608}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2304740608}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:03.778 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24549:up:rejoin 2021-11-20T06:08:04.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:04 smithi178 conmon[36600]: audit 2021-11-20T06:08:03.780871+0000 mon.smithi139 (mon.0) 875 : audit [DBG] from='client.? 172.21.15.139:0/2575779243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:04.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:04 smithi139 conmon[33501]: audit 2021-11-20T06:08:03.780871+0000 mon.smithi139 (mon.0 2021-11-20T06:08:04.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:04 smithi139 conmon[33501]: ) 875 : audit [DBG] from='client.? 172.21.15.139:0/2575779243' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:04.779 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:04.899 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.912+0000 7f462607e700 1 -- 172.21.15.139:0/3532603714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46200f7920 msgr2=0x7f46200f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:04.900 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.912+0000 7f462607e700 1 --2- 172.21.15.139:0/3532603714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46200f7920 0x7f46200f7d80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f4614004660 tx=0x7f4614009b30).stop 2021-11-20T06:08:04.900 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.912+0000 7f462607e700 1 -- 172.21.15.139:0/3532603714 shutdown_connections 2021-11-20T06:08:04.900 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.912+0000 7f462607e700 1 --2- 172.21.15.139:0/3532603714 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46200f7920 0x7f46200f7d80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:04.900 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.912+0000 7f462607e700 1 --2- 172.21.15.139:0/3532603714 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46200f6680 0x7f46200f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:08:04.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.912+0000 7f462607e700 1 -- 172.21.15.139:0/3532603714 >> 172.21.15.139:0/3532603714 conn(0x7f46200f1de0 msgr2=0x7f46200f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:04.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.912+0000 7f462607e700 1 -- 172.21.15.139:0/3532603714 shutdown_connections 2021-11-20T06:08:04.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.912+0000 7f462607e700 1 -- 172.21.15.139:0/3532603714 wait complete. 2021-11-20T06:08:04.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.912+0000 7f462607e700 1 Processor -- start 2021-11-20T06:08:04.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.912+0000 7f462607e700 1 -- start start 2021-11-20T06:08:04.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.912+0000 7f462607e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46200f6680 0x7f4620106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:04.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.912+0000 7f462607e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46200f7920 0x7f46201073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:04.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.912+0000 7f462607e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46201079e0 con 0x7f46200f6680 2021-11-20T06:08:04.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.912+0000 7f462607e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4620107b20 con 0x7f46200f7920 2021-11-20T06:08:04.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.913+0000 7f461f7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46200f6680 0x7f4620106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:04.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.913+0000 7f461f7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46200f6680 0x7f4620106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59082/0 (socket says 172.21.15.139:59082) 2021-11-20T06:08:04.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.913+0000 7f461f7fe700 1 -- 172.21.15.139:0/4183916181 learned_addr learned my addr 172.21.15.139:0/4183916181 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:04.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.913+0000 7f461effd700 1 --2- 172.21.15.139:0/4183916181 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46200f7920 0x7f46201073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:04.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.913+0000 7f461f7fe700 1 -- 172.21.15.139:0/4183916181 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46200f7920 msgr2=0x7f46201073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:04.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.913+0000 7f461f7fe700 1 --2- 172.21.15.139:0/4183916181 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46200f7920 0x7f46201073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:04.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.913+0000 7f461f7fe700 1 -- 172.21.15.139:0/4183916181 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4614005040 con 0x7f46200f6680 2021-11-20T06:08:04.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.913+0000 7f461effd700 1 --2- 172.21.15.139:0/4183916181 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46200f7920 0x7f46201073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:08:04.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.913+0000 7f461f7fe700 1 --2- 172.21.15.139:0/4183916181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46200f6680 0x7f4620106eb0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f46100093f0 tx=0x7f4610006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:04.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.913+0000 7f4607fff700 1 -- 172.21.15.139:0/4183916181 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4610013640 con 0x7f46200f6680 2021-11-20T06:08:04.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.913+0000 7f4607fff700 1 -- 172.21.15.139:0/4183916181 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4610009e40 con 0x7f46200f6680 2021-11-20T06:08:04.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.913+0000 7f462607e700 1 -- 172.21.15.139:0/4183916181 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f462010c5c0 con 0x7f46200f6680 2021-11-20T06:08:04.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.913+0000 7f4607fff700 1 -- 172.21.15.139:0/4183916181 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f461000dbf0 con 0x7f46200f6680 2021-11-20T06:08:04.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.913+0000 7f462607e700 1 -- 172.21.15.139:0/4183916181 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462010cb80 con 0x7f46200f6680 2021-11-20T06:08:04.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.914+0000 7f462607e700 1 -- 172.21.15.139:0/4183916181 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4620027a00 con 0x7f46200f6680 2021-11-20T06:08:04.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.914+0000 7f4607fff700 1 -- 172.21.15.139:0/4183916181 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f461001a060 con 0x7f46200f6680 2021-11-20T06:08:04.907 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.915+0000 7f4607fff700 1 --2- 172.21.15.139:0/4183916181 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f460805c2a0 0x7f460805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:04.907 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.915+0000 7f4607fff700 1 -- 172.21.15.139:0/4183916181 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f46100764b0 con 0x7f46200f6680 2021-11-20T06:08:04.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.915+0000 7f461effd700 1 --2- 172.21.15.139:0/4183916181 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f460805c2a0 0x7f460805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:04.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.915+0000 7f461effd700 1 --2- 172.21.15.139:0/4183916181 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f460805c2a0 0x7f460805e760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f4614013580 tx=0x7f4614009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:04.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:04.918+0000 7f4607fff700 1 -- 172.21.15.139:0/4183916181 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4610048bb0 con 0x7f46200f6680 2021-11-20T06:08:05.087 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:05.101+0000 7f462607e700 1 -- 172.21.15.139:0/4183916181 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46200feeb0 con 0x7f46200f6680 2021-11-20T06:08:05.088 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:05.102+0000 7f4607fff700 1 -- 172.21.15.139:0/4183916181 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f461004c1d0 con 0x7f46200f6680 2021-11-20T06:08:05.088 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:05.089 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:02.313577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24549},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24549":{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:05.090 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:05.103+0000 7f462607e700 1 -- 172.21.15.139:0/4183916181 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f460805c2a0 msgr2=0x7f460805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:05.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:05.103+0000 7f462607e700 1 --2- 172.21.15.139:0/4183916181 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f460805c2a0 0x7f460805e760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f4614013580 tx=0x7f4614009b30).stop 2021-11-20T06:08:05.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:05.103+0000 7f462607e700 1 -- 172.21.15.139:0/4183916181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46200f6680 msgr2=0x7f4620106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:05.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:05.103+0000 7f462607e700 1 --2- 172.21.15.139:0/4183916181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46200f6680 0x7f4620106eb0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f46100093f0 tx=0x7f4610006ec0).stop 2021-11-20T06:08:05.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:05.103+0000 7f462607e700 1 -- 172.21.15.139:0/4183916181 shutdown_connections 2021-11-20T06:08:05.091 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:05.103+0000 7f462607e700 1 --2- 172.21.15.139:0/4183916181 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f460805c2a0 0x7f460805e760 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:05.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:05.103+0000 7f462607e700 1 --2- 172.21.15.139:0/4183916181 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46200f6680 0x7f4620106eb0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:05.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:05.103+0000 7f462607e700 1 --2- 172.21.15.139:0/4183916181 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46200f7920 0x7f46201073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:05.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:05.103+0000 7f462607e700 1 -- 172.21.15.139:0/4183916181 >> 172.21.15.139:0/4183916181 conn(0x7f46200f1de0 msgr2=0x7f46200fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:05.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:05.103+0000 7f462607e700 1 -- 172.21.15.139:0/4183916181 shutdown_connections 2021-11-20T06:08:05.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:05.103+0000 7f462607e700 1 -- 172.21.15.139:0/4183916181 wait complete. 2021-11-20T06:08:05.095 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 87 2021-11-20T06:08:05.101 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:02.313577+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24549}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24549': {'gid': 24549, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 84, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6827/2304740608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2304740608}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2304740608}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:05.102 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24549:up:rejoin 2021-11-20T06:08:05.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:05 smithi139 conmon[33501]: cluster 2021-11-20T06:08:04.279567+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:08:05.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:05 smithi139 conmon[33501]: 328 : cluster [DBG] pgmap v305: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T06:08:05.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:05 smithi139 conmon[33501]: cluster 2021-11-20T06:08:04.329783+0000 mon.smithi139 (mon.0) 876 : cluster [DBG] mds.? [v2:172.21.15.178:6824/4081910640,v1:172.21.15.178:6825/4081910640] up:standby 2021-11-20T06:08:05.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:05 smithi139 conmon[33501]: cluster 2021-11-20T06:08:04.329826+0000 mon.smithi139 ( 2021-11-20T06:08:05.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:05 smithi139 conmon[33501]: mon.0) 877 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:rejoin} 1 up:standby 2021-11-20T06:08:05.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:05 smithi139 conmon[33501]: audit 2021-11-20T06:08:05.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:05 smithi139 conmon[33501]: 2021-11-20T06:08: 2021-11-20T06:08:05.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:05 smithi139 conmon[33501]: 05.103543+0000 mon.smithi139 (mon.0) 878 : 2021-11-20T06:08:05.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:05 smithi139 conmon[33501]: audit [DBG] from='client.? 172.21.15.139:0/4183916181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:05.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:05 smithi178 conmon[36600]: cluster 2021-11-20T06:08:04.279567+0000 mgr.smithi139.dplyxn (mgr.14162) 328 : cluster [DBG] pgmap v305: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-20T06:08:05.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:05 smithi178 conmon[36600]: cluster 2021-11-20T06:08:04.329783+0000 mon.smithi139 (mon.0) 876 : cluster [DBG] mds.? [v2:172.21.15.178:6824/4081910640,v1:172.21.15.178:6825/4081910640] up:standby 2021-11-20T06:08:05.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:05 smithi178 conmon[36600]: cluster 2021-11-20T06:08:04.329826+0000 mon.smithi139 (mon.0) 877 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.pskihy=up:rejoin} 1 up:standby 2021-11-20T06:08:05.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:05 smithi178 conmon[36600]: audit 2021-11-20T06: 2021-11-20T06:08:05.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:05 smithi178 conmon[36600]: 08:05.103543+0000 mon.smithi139 (mon.0 2021-11-20T06:08:05.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:05 smithi178 conmon[36600]: ) 878 : audit [DBG] from='client.? 172.21.15.139:0/4183916181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:06.103 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:06.233 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.246+0000 7f207908d700 1 -- 172.21.15.139:0/2127210099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20740f6820 msgr2=0x7f20740f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:06.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.246+0000 7f207908d700 1 --2- 172.21.15.139:0/2127210099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20740f6820 0x7f20740f6c80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f2068004660 tx=0x7f2068009b30).stop 2021-11-20T06:08:06.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.246+0000 7f207908d700 1 -- 172.21.15.139:0/2127210099 shutdown_connections 2021-11-20T06:08:06.234 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.246+0000 7f207908d700 1 --2- 172.21.15.139:0/2127210099 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20740f6820 0x7f20740f6c80 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:06.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.246+0000 7f207908d700 1 --2- 172.21.15.139:0/2127210099 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20740f5580 0x7f20740f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:06.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.246+0000 7f207908d700 1 -- 172.21.15.139:0/2127210099 >> 172.21.15.139:0/2127210099 conn(0x7f20740f0ce0 msgr2=0x7f20740f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:06.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.246+0000 7f207908d700 1 -- 172.21.15.139:0/2127210099 shutdown_connections 2021-11-20T06:08:06.235 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.246+0000 7f207908d700 1 -- 172.21.15.139:0/2127210099 wait complete. 2021-11-20T06:08:06.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.246+0000 7f207908d700 1 Processor -- start 2021-11-20T06:08:06.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f207908d700 1 -- start start 2021-11-20T06:08:06.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f207908d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20740f5580 0x7f2074105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:06.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f207908d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20740f6820 0x7f2074106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:06.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f207908d700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2074106890 con 0x7f20740f6820 2021-11-20T06:08:06.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f207908d700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20741069d0 con 0x7f20740f5580 2021-11-20T06:08:06.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f207259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20740f6820 0x7f2074106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:06.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f207259c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20740f6820 0x7f2074106290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59096/0 (socket says 172.21.15.139:59096) 2021-11-20T06:08:06.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f207259c700 1 -- 172.21.15.139:0/3645979703 learned_addr learned my addr 172.21.15.139:0/3645979703 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:06.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f2072d9d700 1 --2- 172.21.15.139:0/3645979703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20740f5580 0x7f2074105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:06.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f207259c700 1 -- 172.21.15.139:0/3645979703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20740f5580 msgr2=0x7f2074105da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:06.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f207259c700 1 --2- 172.21.15.139:0/3645979703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20740f5580 0x7f2074105da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:06.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f207259c700 1 -- 172.21.15.139:0/3645979703 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2068005040 con 0x7f20740f6820 2021-11-20T06:08:06.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f207259c700 1 --2- 172.21.15.139:0/3645979703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20740f6820 0x7f2074106290 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f2068009830 tx=0x7f2068009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:06.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f205b7fe700 1 -- 172.21.15.139:0/3645979703 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2068018070 con 0x7f20740f6820 2021-11-20T06:08:06.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f205b7fe700 1 -- 172.21.15.139:0/3645979703 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2068013410 con 0x7f20740f6820 2021-11-20T06:08:06.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f207908d700 1 -- 172.21.15.139:0/3645979703 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f207410b380 con 0x7f20740f6820 2021-11-20T06:08:06.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f205b7fe700 1 -- 172.21.15.139:0/3645979703 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f206801c620 con 0x7f20740f6820 2021-11-20T06:08:06.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.247+0000 7f207908d700 1 -- 172.21.15.139:0/3645979703 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f207410b8e0 con 0x7f20740f6820 2021-11-20T06:08:06.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.248+0000 7f207908d700 1 -- 172.21.15.139:0/3645979703 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2074027a00 con 0x7f20740f6820 2021-11-20T06:08:06.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.249+0000 7f205b7fe700 1 -- 172.21.15.139:0/3645979703 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f206801c8e0 con 0x7f20740f6820 2021-11-20T06:08:06.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.249+0000 7f205b7fe700 1 --2- 172.21.15.139:0/3645979703 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f205c05c2a0 0x7f205c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:06.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.249+0000 7f205b7fe700 1 -- 172.21.15.139:0/3645979703 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f2068078300 con 0x7f20740f6820 2021-11-20T06:08:06.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.249+0000 7f2072d9d700 1 --2- 172.21.15.139:0/3645979703 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f205c05c2a0 0x7f205c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:06.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.249+0000 7f2072d9d700 1 --2- 172.21.15.139:0/3645979703 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f205c05c2a0 0x7f205c05e760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f2074107290 tx=0x7f2064009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:06.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.253+0000 7f205b7fe700 1 -- 172.21.15.139:0/3645979703 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f206804aa00 con 0x7f20740f6820 2021-11-20T06:08:06.424 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.438+0000 7f207908d700 1 -- 172.21.15.139:0/3645979703 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20740fdd10 con 0x7f20740f6820 2021-11-20T06:08:06.424 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.439+0000 7f205b7fe700 1 -- 172.21.15.139:0/3645979703 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f206804e020 con 0x7f20740f6820 2021-11-20T06:08:06.425 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:06.425 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:02.313577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24549},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24549":{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:06.427 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.440+0000 7f207908d700 1 -- 172.21.15.139:0/3645979703 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f205c05c2a0 msgr2=0x7f205c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:06.427 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.440+0000 7f207908d700 1 --2- 172.21.15.139:0/3645979703 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f205c05c2a0 0x7f205c05e760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f2074107290 tx=0x7f2064009040).stop 2021-11-20T06:08:06.427 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.440+0000 7f207908d700 1 -- 172.21.15.139:0/3645979703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20740f6820 msgr2=0x7f2074106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:06.427 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.440+0000 7f207908d700 1 --2- 172.21.15.139:0/3645979703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20740f6820 0x7f2074106290 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f2068009830 tx=0x7f2068009b30).stop 2021-11-20T06:08:06.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.440+0000 7f207908d700 1 -- 172.21.15.139:0/3645979703 shutdown_connections 2021-11-20T06:08:06.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.440+0000 7f207908d700 1 --2- 172.21.15.139:0/3645979703 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f205c05c2a0 0x7f205c05e760 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:06.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.440+0000 7f207908d700 1 --2- 172.21.15.139:0/3645979703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20740f5580 0x7f2074105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:06.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.440+0000 7f207908d700 1 --2- 172.21.15.139:0/3645979703 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20740f6820 0x7f2074106290 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:06.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.440+0000 7f207908d700 1 -- 172.21.15.139:0/3645979703 >> 172.21.15.139:0/3645979703 conn(0x7f20740f0ce0 msgr2=0x7f20740fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:06.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.441+0000 7f207908d700 1 -- 172.21.15.139:0/3645979703 shutdown_connections 2021-11-20T06:08:06.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:06.441+0000 7f207908d700 1 -- 172.21.15.139:0/3645979703 wait complete. 2021-11-20T06:08:06.429 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 87 2021-11-20T06:08:06.438 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:02.313577+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24549}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24549': {'gid': 24549, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 84, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6827/2304740608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2304740608}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2304740608}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:06.438 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24549:up:rejoin 2021-11-20T06:08:07.439 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:07.467 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:07 smithi139 conmon[33501]: cluster 2021-11-20T06:08:06.279946+0000 mgr.smithi139.dplyxn (mgr.14162) 329 : 2021-11-20T06:08:07.467 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:07 smithi139 conmon[33501]: cluster [DBG] pgmap v306: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:08:07.467 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:07 smithi139 conmon[33501]: audit 2021-11-20T06:08:06.440761+0000 mon.smithi139 (mon.0) 879 : audit [DBG] from='client.? 172.21.15.139:0/3645979703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:07.568 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.579+0000 7fd7ee8ad700 1 -- 172.21.15.139:0/2711491109 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e80f65d0 msgr2=0x7fd7e80f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:07.568 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.579+0000 7fd7ee8ad700 1 --2- 172.21.15.139:0/2711491109 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e80f65d0 0x7fd7e80f69f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fd7e4004660 tx=0x7fd7e4009b30).stop 2021-11-20T06:08:07.568 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.579+0000 7fd7ee8ad700 1 -- 172.21.15.139:0/2711491109 shutdown_connections 2021-11-20T06:08:07.569 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.579+0000 7fd7ee8ad700 1 --2- 172.21.15.139:0/2711491109 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd7e80f7870 0x7fd7e80f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:07.569 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.579+0000 7fd7ee8ad700 1 --2- 172.21.15.139:0/2711491109 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e80f65d0 0x7fd7e80f69f0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:07.569 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.579+0000 7fd7ee8ad700 1 -- 172.21.15.139:0/2711491109 >> 172.21.15.139:0/2711491109 conn(0x7fd7e80f1d10 msgr2=0x7fd7e80f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:07.569 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.579+0000 7fd7ee8ad700 1 -- 172.21.15.139:0/2711491109 shutdown_connections 2021-11-20T06:08:07.569 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.579+0000 7fd7ee8ad700 1 -- 172.21.15.139:0/2711491109 wait complete. 2021-11-20T06:08:07.570 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.580+0000 7fd7ee8ad700 1 Processor -- start 2021-11-20T06:08:07.570 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.580+0000 7fd7ee8ad700 1 -- start start 2021-11-20T06:08:07.570 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.580+0000 7fd7ee8ad700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd7e80f65d0 0x7fd7e8106da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:07.570 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.580+0000 7fd7ee8ad700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e80f7870 0x7fd7e8107290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:07.571 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.580+0000 7fd7ee8ad700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e8107890 con 0x7fd7e80f7870 2021-11-20T06:08:07.571 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.580+0000 7fd7ee8ad700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e81080d0 con 0x7fd7e80f65d0 2021-11-20T06:08:07.571 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.580+0000 7fd7ed0aa700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e80f7870 0x7fd7e8107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:07.571 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.580+0000 7fd7ed0aa700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e80f7870 0x7fd7e8107290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59108/0 (socket says 172.21.15.139:59108) 2021-11-20T06:08:07.572 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.580+0000 7fd7ed0aa700 1 -- 172.21.15.139:0/3293600643 learned_addr learned my addr 172.21.15.139:0/3293600643 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:07.572 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.580+0000 7fd7ed0aa700 1 -- 172.21.15.139:0/3293600643 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd7e80f65d0 msgr2=0x7fd7e8106da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:07.572 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.580+0000 7fd7ed8ab700 1 --2- 172.21.15.139:0/3293600643 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd7e80f65d0 0x7fd7e8106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:07.572 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.580+0000 7fd7ed0aa700 1 --2- 172.21.15.139:0/3293600643 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd7e80f65d0 0x7fd7e8106da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:07.572 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.580+0000 7fd7ed0aa700 1 -- 172.21.15.139:0/3293600643 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e4005040 con 0x7fd7e80f7870 2021-11-20T06:08:07.573 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.581+0000 7fd7ed0aa700 1 --2- 172.21.15.139:0/3293600643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e80f7870 0x7fd7e8107290 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7fd7d800c530 tx=0x7fd7d8007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:07.573 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.581+0000 7fd7de7fc700 1 -- 172.21.15.139:0/3293600643 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7d800f650 con 0x7fd7e80f7870 2021-11-20T06:08:07.573 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.581+0000 7fd7de7fc700 1 -- 172.21.15.139:0/3293600643 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7d800fc60 con 0x7fd7e80f7870 2021-11-20T06:08:07.573 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.581+0000 7fd7ee8ad700 1 -- 172.21.15.139:0/3293600643 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e81083b0 con 0x7fd7e80f7870 2021-11-20T06:08:07.573 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.581+0000 7fd7de7fc700 1 -- 172.21.15.139:0/3293600643 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7d8012c80 con 0x7fd7e80f7870 2021-11-20T06:08:07.574 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.581+0000 7fd7ee8ad700 1 -- 172.21.15.139:0/3293600643 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e81088f0 con 0x7fd7e80f7870 2021-11-20T06:08:07.574 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.582+0000 7fd7ee8ad700 1 -- 172.21.15.139:0/3293600643 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e80f59c0 con 0x7fd7e80f7870 2021-11-20T06:08:07.574 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.582+0000 7fd7de7fc700 1 -- 172.21.15.139:0/3293600643 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd7d8010070 con 0x7fd7e80f7870 2021-11-20T06:08:07.574 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.582+0000 7fd7de7fc700 1 --2- 172.21.15.139:0/3293600643 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd7d405c300 0x7fd7d405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:07.575 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.582+0000 7fd7de7fc700 1 -- 172.21.15.139:0/3293600643 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fd7d8076160 con 0x7fd7e80f7870 2021-11-20T06:08:07.575 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.583+0000 7fd7ed8ab700 1 --2- 172.21.15.139:0/3293600643 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd7d405c300 0x7fd7d405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:07.575 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.583+0000 7fd7ed8ab700 1 --2- 172.21.15.139:0/3293600643 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd7d405c300 0x7fd7d405e7c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fd7e4009d60 tx=0x7fd7e4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:07.576 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.587+0000 7fd7de7fc700 1 -- 172.21.15.139:0/3293600643 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7d80452b0 con 0x7fd7e80f7870 2021-11-20T06:08:07.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:07 smithi178 conmon[36600]: cluster 2021-11-20T06:08:06.279946+0000 mgr.smithi139.dplyxn (mgr.14162) 329 : cluster [DBG] pgmap v306: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:08:07.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:07 smithi178 conmon[36600]: audit 2021-11-20T06:08:06.440761+0000 mon.smithi139 (mon.0) 879 : audit [DBG] from='client.? 172.21.15.139:0/3645979703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:07.755 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.769+0000 7fd7ee8ad700 1 -- 172.21.15.139:0/3293600643 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7e80fedf0 con 0x7fd7e80f7870 2021-11-20T06:08:07.756 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.769+0000 7fd7de7fc700 1 -- 172.21.15.139:0/3293600643 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2187 (secure 0 0 0) 0x7fd7d804c2f0 con 0x7fd7e80f7870 2021-11-20T06:08:07.756 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:07.756 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:02.313577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24549},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24549":{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:07.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.771+0000 7fd7ee8ad700 1 -- 172.21.15.139:0/3293600643 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd7d405c300 msgr2=0x7fd7d405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:07.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.771+0000 7fd7ee8ad700 1 --2- 172.21.15.139:0/3293600643 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd7d405c300 0x7fd7d405e7c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fd7e4009d60 tx=0x7fd7e4009b30).stop 2021-11-20T06:08:07.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.771+0000 7fd7ee8ad700 1 -- 172.21.15.139:0/3293600643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e80f7870 msgr2=0x7fd7e8107290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:07.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.771+0000 7fd7ee8ad700 1 --2- 172.21.15.139:0/3293600643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e80f7870 0x7fd7e8107290 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7fd7d800c530 tx=0x7fd7d8007e20).stop 2021-11-20T06:08:07.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.771+0000 7fd7ee8ad700 1 -- 172.21.15.139:0/3293600643 shutdown_connections 2021-11-20T06:08:07.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.771+0000 7fd7ee8ad700 1 --2- 172.21.15.139:0/3293600643 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fd7d405c300 0x7fd7d405e7c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:07.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.771+0000 7fd7ee8ad700 1 --2- 172.21.15.139:0/3293600643 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd7e80f65d0 0x7fd7e8106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:07.759 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.771+0000 7fd7ee8ad700 1 --2- 172.21.15.139:0/3293600643 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e80f7870 0x7fd7e8107290 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:07.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.771+0000 7fd7ee8ad700 1 -- 172.21.15.139:0/3293600643 >> 172.21.15.139:0/3293600643 conn(0x7fd7e80f1d10 msgr2=0x7fd7e80fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:07.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.771+0000 7fd7ee8ad700 1 -- 172.21.15.139:0/3293600643 shutdown_connections 2021-11-20T06:08:07.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:07.771+0000 7fd7ee8ad700 1 -- 172.21.15.139:0/3293600643 wait complete. 2021-11-20T06:08:07.762 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 87 2021-11-20T06:08:07.769 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:02.313577+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24549}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24549': {'gid': 24549, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 84, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6827/2304740608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2304740608}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2304740608}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:07.771 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24549:up:rejoin 2021-11-20T06:08:08.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:08 smithi178 conmon[36600]: audit 2021-11-20T06:08:07.771063+0000 mon.smithi139 (mon.0) 880 : audit [DBG] from='client.? 172.21.15.139:0/3293600643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:08.773 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:08.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:08 smithi139 conmon[33501]: audit 2021-11-20T06:08:07. 2021-11-20T06:08:08.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:08 smithi139 conmon[33501]: 771063+0000 mon.smithi139 (mon.0) 880 : audit [DBG] from='client.? 172.21.15.139:0/3293600643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:08.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.913+0000 7f3ac4eb2700 1 -- 172.21.15.139:0/3232030029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ac00f5590 msgr2=0x7f3ac00f59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:08.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.913+0000 7f3ac4eb2700 1 --2- 172.21.15.139:0/3232030029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ac00f5590 0x7f3ac00f59b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f3ab0004660 tx=0x7f3ab0009b30).stop 2021-11-20T06:08:08.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.913+0000 7f3ac4eb2700 1 -- 172.21.15.139:0/3232030029 shutdown_connections 2021-11-20T06:08:08.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.913+0000 7f3ac4eb2700 1 --2- 172.21.15.139:0/3232030029 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ac00f6830 0x7f3ac00f6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:08.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.913+0000 7f3ac4eb2700 1 --2- 172.21.15.139:0/3232030029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ac00f5590 0x7f3ac00f59b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:08.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.913+0000 7f3ac4eb2700 1 -- 172.21.15.139:0/3232030029 >> 172.21.15.139:0/3232030029 conn(0x7f3ac00f0cd0 msgr2=0x7f3ac00f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:08.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.913+0000 7f3ac4eb2700 1 -- 172.21.15.139:0/3232030029 shutdown_connections 2021-11-20T06:08:08.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.913+0000 7f3ac4eb2700 1 -- 172.21.15.139:0/3232030029 wait complete. 2021-11-20T06:08:08.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3ac4eb2700 1 Processor -- start 2021-11-20T06:08:08.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3ac4eb2700 1 -- start start 2021-11-20T06:08:08.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3ac4eb2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ac00f5590 0x7f3ac0060530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:08.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3ac4eb2700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ac00f6830 0x7f3ac010bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:08.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3ac4eb2700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac010c1a0 con 0x7f3ac00f5590 2021-11-20T06:08:08.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3ac4eb2700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac010c2e0 con 0x7f3ac00f6830 2021-11-20T06:08:08.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3abe59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ac00f5590 0x7f3ac0060530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:08.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3abe59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ac00f5590 0x7f3ac0060530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59116/0 (socket says 172.21.15.139:59116) 2021-11-20T06:08:08.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3abe59c700 1 -- 172.21.15.139:0/2399448626 learned_addr learned my addr 172.21.15.139:0/2399448626 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:08.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3abe59c700 1 -- 172.21.15.139:0/2399448626 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ac00f6830 msgr2=0x7f3ac010bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:08.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3abe59c700 1 --2- 172.21.15.139:0/2399448626 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ac00f6830 0x7f3ac010bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:08:08.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3abe59c700 1 -- 172.21.15.139:0/2399448626 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab0005040 con 0x7f3ac00f5590 2021-11-20T06:08:08.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3abe59c700 1 --2- 172.21.15.139:0/2399448626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ac00f5590 0x7f3ac0060530 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f3ab0009f50 tx=0x7f3ab0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:08.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3aaeffd700 1 -- 172.21.15.139:0/2399448626 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ab0018070 con 0x7f3ac00f5590 2021-11-20T06:08:08.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3aaeffd700 1 -- 172.21.15.139:0/2399448626 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ab0013410 con 0x7f3ac00f5590 2021-11-20T06:08:08.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3ac4eb2700 1 -- 172.21.15.139:0/2399448626 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac010c4e0 con 0x7f3ac00f5590 2021-11-20T06:08:08.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.914+0000 7f3aaeffd700 1 -- 172.21.15.139:0/2399448626 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ab001de70 con 0x7f3ac00f5590 2021-11-20T06:08:08.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.915+0000 7f3ac4eb2700 1 -- 172.21.15.139:0/2399448626 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac010ca40 con 0x7f3ac00f5590 2021-11-20T06:08:08.907 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.915+0000 7f3ac4eb2700 1 -- 172.21.15.139:0/2399448626 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac0027a00 con 0x7f3ac00f5590 2021-11-20T06:08:08.907 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.916+0000 7f3aaeffd700 1 -- 172.21.15.139:0/2399448626 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3ab001d420 con 0x7f3ac00f5590 2021-11-20T06:08:08.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.916+0000 7f3aaeffd700 1 --2- 172.21.15.139:0/2399448626 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3aa805c2a0 0x7f3aa805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:08.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.916+0000 7f3aaeffd700 1 -- 172.21.15.139:0/2399448626 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f3ab0078040 con 0x7f3ac00f5590 2021-11-20T06:08:08.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.916+0000 7f3abdd9b700 1 --2- 172.21.15.139:0/2399448626 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3aa805c2a0 0x7f3aa805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:08.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.916+0000 7f3abdd9b700 1 --2- 172.21.15.139:0/2399448626 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3aa805c2a0 0x7f3aa805e760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3ab4004a10 tx=0x7f3ab40099e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:08.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:08.920+0000 7f3aaeffd700 1 -- 172.21.15.139:0/2399448626 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ab004a7a0 con 0x7f3ac00f5590 2021-11-20T06:08:09.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:09.107+0000 7f3ac4eb2700 1 -- 172.21.15.139:0/2399448626 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ac00fdd20 con 0x7f3ac00f5590 2021-11-20T06:08:09.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:09.107+0000 7f3aaeffd700 1 -- 172.21.15.139:0/2399448626 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f3ab004ddc0 con 0x7f3ac00f5590 2021-11-20T06:08:09.093 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:09.095 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:02.313577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24549},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24549":{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:09.096 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:09.109+0000 7f3ac4eb2700 1 -- 172.21.15.139:0/2399448626 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3aa805c2a0 msgr2=0x7f3aa805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:09.097 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:09.109+0000 7f3ac4eb2700 1 --2- 172.21.15.139:0/2399448626 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3aa805c2a0 0x7f3aa805e760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3ab4004a10 tx=0x7f3ab40099e0).stop 2021-11-20T06:08:09.097 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:09.109+0000 7f3ac4eb2700 1 -- 172.21.15.139:0/2399448626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ac00f5590 msgr2=0x7f3ac0060530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:09.097 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:09.109+0000 7f3ac4eb2700 1 --2- 172.21.15.139:0/2399448626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ac00f5590 0x7f3ac0060530 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f3ab0009f50 tx=0x7f3ab0009b30).stop 2021-11-20T06:08:09.098 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:09.109+0000 7f3ac4eb2700 1 -- 172.21.15.139:0/2399448626 shutdown_connections 2021-11-20T06:08:09.098 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:09.109+0000 7f3ac4eb2700 1 --2- 172.21.15.139:0/2399448626 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f3aa805c2a0 0x7f3aa805e760 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:09.098 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:09.109+0000 7f3ac4eb2700 1 --2- 172.21.15.139:0/2399448626 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3ac00f5590 0x7f3ac0060530 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:09.099 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:09.109+0000 7f3ac4eb2700 1 --2- 172.21.15.139:0/2399448626 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ac00f6830 0x7f3ac010bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:08:09.099 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:09.109+0000 7f3ac4eb2700 1 -- 172.21.15.139:0/2399448626 >> 172.21.15.139:0/2399448626 conn(0x7f3ac00f0cd0 msgr2=0x7f3ac00fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:09.099 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:09.109+0000 7f3ac4eb2700 1 -- 172.21.15.139:0/2399448626 shutdown_connections 2021-11-20T06:08:09.099 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:09.109+0000 7f3ac4eb2700 1 -- 172.21.15.139:0/2399448626 wait complete. 2021-11-20T06:08:09.099 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 87 2021-11-20T06:08:09.107 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:02.313577+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24549}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24549': {'gid': 24549, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 84, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6827/2304740608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2304740608}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2304740608}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:09.107 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24549:up:rejoin 2021-11-20T06:08:09.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:09 smithi139 conmon[33501]: cluster 2021-11-20T06:08:08. 2021-11-20T06:08:09.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:09 smithi139 conmon[33501]: 280465+0000 mgr.smithi139.dplyxn (mgr.14162) 330 : cluster [DBG] pgmap v307: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:08:09.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:09 smithi139 conmon[33501]: audit 2021-11-20T06:08:09.109044 2021-11-20T06:08:09.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:09 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 881 : audit [DBG] from='client.? 172.21.15.139:0/2399448626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:09.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:09 smithi178 conmon[36600]: cluster 2021-11-20T06:08:08.280465+0000 mgr.smithi139.dplyxn (mgr.14162) 330 : cluster [DBG] pgmap v307: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:08:09.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:09 smithi178 conmon[36600]: audit 2021-11-20T06:08:09.109044+0000 mon.smithi139 (mon.0) 881 : audit [DBG] from='client.? 172.21.15.139:0/2399448626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:10.108 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:10.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.250+0000 7f7151a5c700 1 -- 172.21.15.139:0/4221394208 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f714c0f7920 msgr2=0x7f714c0f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:10.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.250+0000 7f7151a5c700 1 --2- 172.21.15.139:0/4221394208 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f714c0f7920 0x7f714c0f7d80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f7140004660 tx=0x7f7140009b30).stop 2021-11-20T06:08:10.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.250+0000 7f7151a5c700 1 -- 172.21.15.139:0/4221394208 shutdown_connections 2021-11-20T06:08:10.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.250+0000 7f7151a5c700 1 --2- 172.21.15.139:0/4221394208 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f714c0f7920 0x7f714c0f7d80 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:10.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.250+0000 7f7151a5c700 1 --2- 172.21.15.139:0/4221394208 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f714c0f6680 0x7f714c0f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:10.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.250+0000 7f7151a5c700 1 -- 172.21.15.139:0/4221394208 >> 172.21.15.139:0/4221394208 conn(0x7f714c0f1de0 msgr2=0x7f714c0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:10.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.250+0000 7f7151a5c700 1 -- 172.21.15.139:0/4221394208 shutdown_connections 2021-11-20T06:08:10.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.250+0000 7f7151a5c700 1 -- 172.21.15.139:0/4221394208 wait complete. 2021-11-20T06:08:10.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.251+0000 7f7151a5c700 1 Processor -- start 2021-11-20T06:08:10.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.251+0000 7f7151a5c700 1 -- start start 2021-11-20T06:08:10.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.251+0000 7f7151a5c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f714c0f6680 0x7f714c106df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:10.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.251+0000 7f7151a5c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f714c0f7920 0x7f714c1072e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:10.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.251+0000 7f7151a5c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c107890 con 0x7f714c0f6680 2021-11-20T06:08:10.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.251+0000 7f7151a5c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c1079d0 con 0x7f714c0f7920 2021-11-20T06:08:10.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.251+0000 7f714affd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f714c0f6680 0x7f714c106df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:10.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.251+0000 7f714affd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f714c0f6680 0x7f714c106df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59124/0 (socket says 172.21.15.139:59124) 2021-11-20T06:08:10.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.251+0000 7f714affd700 1 -- 172.21.15.139:0/3442941421 learned_addr learned my addr 172.21.15.139:0/3442941421 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:10.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.251+0000 7f714affd700 1 -- 172.21.15.139:0/3442941421 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f714c0f7920 msgr2=0x7f714c1072e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:10.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.251+0000 7f714affd700 1 --2- 172.21.15.139:0/3442941421 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f714c0f7920 0x7f714c1072e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:08:10.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.251+0000 7f714affd700 1 -- 172.21.15.139:0/3442941421 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7140005040 con 0x7f714c0f6680 2021-11-20T06:08:10.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.251+0000 7f714affd700 1 --2- 172.21.15.139:0/3442941421 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f714c0f6680 0x7f714c106df0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f713c002700 tx=0x7f713c009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:10.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.252+0000 7f71337fe700 1 -- 172.21.15.139:0/3442941421 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f713c009320 con 0x7f714c0f6680 2021-11-20T06:08:10.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.252+0000 7f71337fe700 1 -- 172.21.15.139:0/3442941421 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f713c00f040 con 0x7f714c0f6680 2021-11-20T06:08:10.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.252+0000 7f7151a5c700 1 -- 172.21.15.139:0/3442941421 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714c10c470 con 0x7f714c0f6680 2021-11-20T06:08:10.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.252+0000 7f71337fe700 1 -- 172.21.15.139:0/3442941421 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f713c0148e0 con 0x7f714c0f6680 2021-11-20T06:08:10.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.252+0000 7f7151a5c700 1 -- 172.21.15.139:0/3442941421 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714c10ca30 con 0x7f714c0f6680 2021-11-20T06:08:10.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.252+0000 7f7151a5c700 1 -- 172.21.15.139:0/3442941421 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714c027a00 con 0x7f714c0f6680 2021-11-20T06:08:10.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.253+0000 7f71337fe700 1 -- 172.21.15.139:0/3442941421 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f713c009480 con 0x7f714c0f6680 2021-11-20T06:08:10.246 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.253+0000 7f71337fe700 1 --2- 172.21.15.139:0/3442941421 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f713405c2f0 0x7f713405e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:10.246 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.253+0000 7f71337fe700 1 -- 172.21.15.139:0/3442941421 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f713c076830 con 0x7f714c0f6680 2021-11-20T06:08:10.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.253+0000 7f714a7fc700 1 --2- 172.21.15.139:0/3442941421 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f713405c2f0 0x7f713405e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:10.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.254+0000 7f714a7fc700 1 --2- 172.21.15.139:0/3442941421 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f713405c2f0 0x7f713405e7b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f7140009f50 tx=0x7f7140009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:10.248 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.257+0000 7f71337fe700 1 -- 172.21.15.139:0/3442941421 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f713c048ed0 con 0x7f714c0f6680 2021-11-20T06:08:10.425 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.440+0000 7f7151a5c700 1 -- 172.21.15.139:0/3442941421 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f714c0fee10 con 0x7f714c0f6680 2021-11-20T06:08:10.426 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.440+0000 7f71337fe700 1 -- 172.21.15.139:0/3442941421 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f713c04c4f0 con 0x7f714c0f6680 2021-11-20T06:08:10.426 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:10.426 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:02.313577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24549},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24549":{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:10.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.443+0000 7f7151a5c700 1 -- 172.21.15.139:0/3442941421 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f713405c2f0 msgr2=0x7f713405e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:10.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.443+0000 7f7151a5c700 1 --2- 172.21.15.139:0/3442941421 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f713405c2f0 0x7f713405e7b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f7140009f50 tx=0x7f7140009b30).stop 2021-11-20T06:08:10.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.443+0000 7f7151a5c700 1 -- 172.21.15.139:0/3442941421 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f714c0f6680 msgr2=0x7f714c106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:10.428 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.443+0000 7f7151a5c700 1 --2- 172.21.15.139:0/3442941421 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f714c0f6680 0x7f714c106df0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f713c002700 tx=0x7f713c009e00).stop 2021-11-20T06:08:10.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.443+0000 7f7151a5c700 1 -- 172.21.15.139:0/3442941421 shutdown_connections 2021-11-20T06:08:10.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.443+0000 7f7151a5c700 1 --2- 172.21.15.139:0/3442941421 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f713405c2f0 0x7f713405e7b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:10.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.443+0000 7f7151a5c700 1 --2- 172.21.15.139:0/3442941421 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f714c0f6680 0x7f714c106df0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:10.429 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.443+0000 7f7151a5c700 1 --2- 172.21.15.139:0/3442941421 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f714c0f7920 0x7f714c1072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:08:10.430 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.443+0000 7f7151a5c700 1 -- 172.21.15.139:0/3442941421 >> 172.21.15.139:0/3442941421 conn(0x7f714c0f1de0 msgr2=0x7f714c0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:10.430 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.443+0000 7f7151a5c700 1 -- 172.21.15.139:0/3442941421 shutdown_connections 2021-11-20T06:08:10.430 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:10.443+0000 7f7151a5c700 1 -- 172.21.15.139:0/3442941421 wait complete. 2021-11-20T06:08:10.430 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 87 2021-11-20T06:08:10.439 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:02.313577+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24549}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24549': {'gid': 24549, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 84, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6827/2304740608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2304740608}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2304740608}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:10.439 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24549:up:rejoin 2021-11-20T06:08:11.440 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:11.573 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.582+0000 7f5ab2a0e700 1 -- 172.21.15.139:0/2344831584 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5aac0f7870 msgr2=0x7f5aac0f7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:11.573 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.582+0000 7f5ab2a0e700 1 --2- 172.21.15.139:0/2344831584 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5aac0f7870 0x7f5aac0f7cd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f5a9c004660 tx=0x7f5a9c009b30).stop 2021-11-20T06:08:11.574 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.582+0000 7f5ab2a0e700 1 -- 172.21.15.139:0/2344831584 shutdown_connections 2021-11-20T06:08:11.574 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.582+0000 7f5ab2a0e700 1 --2- 172.21.15.139:0/2344831584 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5aac0f7870 0x7f5aac0f7cd0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:11.575 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.582+0000 7f5ab2a0e700 1 --2- 172.21.15.139:0/2344831584 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5aac0f65d0 0x7f5aac0f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:11.576 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.582+0000 7f5ab2a0e700 1 -- 172.21.15.139:0/2344831584 >> 172.21.15.139:0/2344831584 conn(0x7f5aac0f1d10 msgr2=0x7f5aac0f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:11.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.583+0000 7f5ab2a0e700 1 -- 172.21.15.139:0/2344831584 shutdown_connections 2021-11-20T06:08:11.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.583+0000 7f5ab2a0e700 1 -- 172.21.15.139:0/2344831584 wait complete. 2021-11-20T06:08:11.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.583+0000 7f5ab2a0e700 1 Processor -- start 2021-11-20T06:08:11.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.583+0000 7f5ab2a0e700 1 -- start start 2021-11-20T06:08:11.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.583+0000 7f5ab2a0e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5aac0f65d0 0x7f5aac06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:11.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.583+0000 7f5ab2a0e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5aac0f7870 0x7f5aac069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:11.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.583+0000 7f5ab2a0e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac06a380 con 0x7f5aac0f7870 2021-11-20T06:08:11.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.583+0000 7f5ab2a0e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac06a4c0 con 0x7f5aac0f65d0 2021-11-20T06:08:11.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.583+0000 7f5ab120b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5aac0f7870 0x7f5aac069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:11.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.583+0000 7f5ab120b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5aac0f7870 0x7f5aac069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59138/0 (socket says 172.21.15.139:59138) 2021-11-20T06:08:11.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.583+0000 7f5ab120b700 1 -- 172.21.15.139:0/402288740 learned_addr learned my addr 172.21.15.139:0/402288740 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:11.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.583+0000 7f5ab1a0c700 1 --2- 172.21.15.139:0/402288740 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5aac0f65d0 0x7f5aac06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:11.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.584+0000 7f5ab120b700 1 -- 172.21.15.139:0/402288740 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5aac0f65d0 msgr2=0x7f5aac06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:11.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.584+0000 7f5ab120b700 1 --2- 172.21.15.139:0/402288740 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5aac0f65d0 0x7f5aac06b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:11.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.584+0000 7f5ab120b700 1 -- 172.21.15.139:0/402288740 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9c005040 con 0x7f5aac0f7870 2021-11-20T06:08:11.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.584+0000 7f5ab1a0c700 1 --2- 172.21.15.139:0/402288740 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5aac0f65d0 0x7f5aac06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:08:11.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.584+0000 7f5ab120b700 1 --2- 172.21.15.139:0/402288740 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5aac0f7870 0x7f5aac069e20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f5a9c009f50 tx=0x7f5a9c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:11.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.584+0000 7f5aa27fc700 1 -- 172.21.15.139:0/402288740 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a9c018070 con 0x7f5aac0f7870 2021-11-20T06:08:11.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.584+0000 7f5aa27fc700 1 -- 172.21.15.139:0/402288740 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a9c013410 con 0x7f5aac0f7870 2021-11-20T06:08:11.582 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.584+0000 7f5aa27fc700 1 -- 172.21.15.139:0/402288740 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a9c01c3c0 con 0x7f5aac0f7870 2021-11-20T06:08:11.582 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.584+0000 7f5ab2a0e700 1 -- 172.21.15.139:0/402288740 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aac06a740 con 0x7f5aac0f7870 2021-11-20T06:08:11.582 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.584+0000 7f5ab2a0e700 1 -- 172.21.15.139:0/402288740 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aac06aca0 con 0x7f5aac0f7870 2021-11-20T06:08:11.582 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.585+0000 7f5ab2a0e700 1 -- 172.21.15.139:0/402288740 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aac045ad0 con 0x7f5aac0f7870 2021-11-20T06:08:11.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.585+0000 7f5aa27fc700 1 -- 172.21.15.139:0/402288740 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5a9c01d420 con 0x7f5aac0f7870 2021-11-20T06:08:11.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.586+0000 7f5aa27fc700 1 --2- 172.21.15.139:0/402288740 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f5a9805c300 0x7f5a9805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:11.584 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.586+0000 7f5aa27fc700 1 -- 172.21.15.139:0/402288740 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f5a9c078040 con 0x7f5aac0f7870 2021-11-20T06:08:11.584 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.586+0000 7f5ab1a0c700 1 --2- 172.21.15.139:0/402288740 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f5a9805c300 0x7f5a9805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:11.584 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.586+0000 7f5ab1a0c700 1 --2- 172.21.15.139:0/402288740 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f5a9805c300 0x7f5a9805e7c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f5aac06b730 tx=0x7f5aa8009c00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:11.585 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.590+0000 7f5aa27fc700 1 -- 172.21.15.139:0/402288740 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a9c04a820 con 0x7f5aac0f7870 2021-11-20T06:08:11.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:11 smithi178 conmon[36600]: cluster 2021-11-20T06:08:10.280884+0000 mgr.smithi139.dplyxn (mgr.14162) 331 : cluster [DBG] pgmap v308: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:08:11.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:11 smithi178 conmon[36600]: audit 2021-11-20T06:08:10.442029+0000 mon.smithi139 (mon.0) 882 : audit [DBG] from='client.? 172.21.15.139:0/3442941421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:11.757 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.773+0000 7f5ab2a0e700 1 -- 172.21.15.139:0/402288740 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5aac027a00 con 0x7f5aac0f7870 2021-11-20T06:08:11.758 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.773+0000 7f5aa27fc700 1 -- 172.21.15.139:0/402288740 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f5a9c04de40 con 0x7f5aac0f7870 2021-11-20T06:08:11.758 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:11.759 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:02.313577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24549},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24549":{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:11.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.775+0000 7f5ab2a0e700 1 -- 172.21.15.139:0/402288740 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f5a9805c300 msgr2=0x7f5a9805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:11.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.775+0000 7f5ab2a0e700 1 --2- 172.21.15.139:0/402288740 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f5a9805c300 0x7f5a9805e7c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f5aac06b730 tx=0x7f5aa8009c00).stop 2021-11-20T06:08:11.760 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.775+0000 7f5ab2a0e700 1 -- 172.21.15.139:0/402288740 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5aac0f7870 msgr2=0x7f5aac069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:11.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.775+0000 7f5ab2a0e700 1 --2- 172.21.15.139:0/402288740 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5aac0f7870 0x7f5aac069e20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f5a9c009f50 tx=0x7f5a9c009b30).stop 2021-11-20T06:08:11.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.775+0000 7f5ab2a0e700 1 -- 172.21.15.139:0/402288740 shutdown_connections 2021-11-20T06:08:11.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.775+0000 7f5ab2a0e700 1 --2- 172.21.15.139:0/402288740 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f5a9805c300 0x7f5a9805e7c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:11.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.775+0000 7f5ab2a0e700 1 --2- 172.21.15.139:0/402288740 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5aac0f65d0 0x7f5aac06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:11.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.775+0000 7f5ab2a0e700 1 --2- 172.21.15.139:0/402288740 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5aac0f7870 0x7f5aac069e20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:11.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.775+0000 7f5ab2a0e700 1 -- 172.21.15.139:0/402288740 >> 172.21.15.139:0/402288740 conn(0x7f5aac0f1d10 msgr2=0x7f5aac0fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:11.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.775+0000 7f5ab2a0e700 1 -- 172.21.15.139:0/402288740 shutdown_connections 2021-11-20T06:08:11.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:11.775+0000 7f5ab2a0e700 1 -- 172.21.15.139:0/402288740 wait complete. 2021-11-20T06:08:11.762 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 87 2021-11-20T06:08:11.770 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:11 smithi139 conmon[33501]: cluster 2021-11-20T06:08:10.280884+0000 mgr.smithi139.dplyxn (mgr.14162) 331 : cluster [DBG] 2021-11-20T06:08:11.770 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:11 smithi139 conmon[33501]: pgmap v308: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:08:11.770 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:11 smithi139 conmon[33501]: audit 2021-11-20T06:08:10.442029+0000 mon.smithi139 (mon.0) 882 : audit [DBG] from='client.? 172.21.15.139:0/3442941421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:11.771 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:02.313577+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24549}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24549': {'gid': 24549, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 84, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6827/2304740608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2304740608}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2304740608}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:11.772 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24549:up:rejoin 2021-11-20T06:08:12.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:12 smithi178 conmon[36600]: audit 2021-11-20T06:08:11.774135+0000 mon.smithi139 (mon.0) 883 : audit [DBG] 2021-11-20T06:08:12.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:12 smithi178 conmon[36600]: from='client.? 172.21.15.139:0/402288740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:12.773 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:12.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:12 smithi139 conmon[33501]: audit 2021-11-20T06:08:11.774135+0000 mon.smithi139 (mon 2021-11-20T06:08:12.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:12 smithi139 conmon[33501]: .0) 883 : audit [DBG] from='client.? 172.21.15.139:0/402288740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:12.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.915+0000 7f1ed396f700 1 -- 172.21.15.139:0/4135065349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ecc0f55a0 msgr2=0x7f1ecc0f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:12.901 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.915+0000 7f1ed396f700 1 --2- 172.21.15.139:0/4135065349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ecc0f55a0 0x7f1ecc0f59c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f1ec80099e0 tx=0x7f1ec8004d10).stop 2021-11-20T06:08:12.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.915+0000 7f1ed396f700 1 -- 172.21.15.139:0/4135065349 shutdown_connections 2021-11-20T06:08:12.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.915+0000 7f1ed396f700 1 --2- 172.21.15.139:0/4135065349 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ecc0f6840 0x7f1ecc0f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:12.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.915+0000 7f1ed396f700 1 --2- 172.21.15.139:0/4135065349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ecc0f55a0 0x7f1ecc0f59c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:12.902 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.915+0000 7f1ed396f700 1 -- 172.21.15.139:0/4135065349 >> 172.21.15.139:0/4135065349 conn(0x7f1ecc0f0ce0 msgr2=0x7f1ecc0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:12.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.915+0000 7f1ed396f700 1 -- 172.21.15.139:0/4135065349 shutdown_connections 2021-11-20T06:08:12.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.915+0000 7f1ed396f700 1 -- 172.21.15.139:0/4135065349 wait complete. 2021-11-20T06:08:12.903 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.915+0000 7f1ed396f700 1 Processor -- start 2021-11-20T06:08:12.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.915+0000 7f1ed396f700 1 -- start start 2021-11-20T06:08:12.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.915+0000 7f1ed396f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ecc0f55a0 0x7f1ecc105eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:12.904 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.915+0000 7f1ed396f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ecc0f6840 0x7f1ecc1063a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:12.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.915+0000 7f1ed396f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc106950 con 0x7f1ecc0f55a0 2021-11-20T06:08:12.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.915+0000 7f1ed396f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc106a90 con 0x7f1ecc0f6840 2021-11-20T06:08:12.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.916+0000 7f1ed170b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ecc0f55a0 0x7f1ecc105eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:12.905 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.916+0000 7f1ed170b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ecc0f55a0 0x7f1ecc105eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59146/0 (socket says 172.21.15.139:59146) 2021-11-20T06:08:12.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.916+0000 7f1ed170b700 1 -- 172.21.15.139:0/3564513379 learned_addr learned my addr 172.21.15.139:0/3564513379 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:12.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.916+0000 7f1ed0f0a700 1 --2- 172.21.15.139:0/3564513379 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ecc0f6840 0x7f1ecc1063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:12.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.916+0000 7f1ed170b700 1 -- 172.21.15.139:0/3564513379 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ecc0f6840 msgr2=0x7f1ecc1063a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:12.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.916+0000 7f1ed170b700 1 --2- 172.21.15.139:0/3564513379 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ecc0f6840 0x7f1ecc1063a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:12.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.916+0000 7f1ed170b700 1 -- 172.21.15.139:0/3564513379 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec8009640 con 0x7f1ecc0f55a0 2021-11-20T06:08:12.907 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.916+0000 7f1ed170b700 1 --2- 172.21.15.139:0/3564513379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ecc0f55a0 0x7f1ecc105eb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f1ec80078d0 tx=0x7f1ec8004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:12.907 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.916+0000 7f1ec1ffb700 1 -- 172.21.15.139:0/3564513379 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ec800e070 con 0x7f1ecc0f55a0 2021-11-20T06:08:12.907 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.916+0000 7f1ec1ffb700 1 -- 172.21.15.139:0/3564513379 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ec8007d70 con 0x7f1ecc0f55a0 2021-11-20T06:08:12.907 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.916+0000 7f1ec1ffb700 1 -- 172.21.15.139:0/3564513379 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ec8018070 con 0x7f1ecc0f55a0 2021-11-20T06:08:12.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.916+0000 7f1ed396f700 1 -- 172.21.15.139:0/3564513379 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ecc10b4d0 con 0x7f1ecc0f55a0 2021-11-20T06:08:12.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.916+0000 7f1ed396f700 1 -- 172.21.15.139:0/3564513379 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ecc10ba90 con 0x7f1ecc0f55a0 2021-11-20T06:08:12.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.917+0000 7f1ed396f700 1 -- 172.21.15.139:0/3564513379 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ecc027a00 con 0x7f1ecc0f55a0 2021-11-20T06:08:12.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.917+0000 7f1ec1ffb700 1 -- 172.21.15.139:0/3564513379 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1ec8024060 con 0x7f1ecc0f55a0 2021-11-20T06:08:12.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.918+0000 7f1ec1ffb700 1 --2- 172.21.15.139:0/3564513379 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1eb805c2a0 0x7f1eb805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:12.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.918+0000 7f1ec1ffb700 1 -- 172.21.15.139:0/3564513379 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f1ec8077a40 con 0x7f1ecc0f55a0 2021-11-20T06:08:12.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.918+0000 7f1ed0f0a700 1 --2- 172.21.15.139:0/3564513379 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1eb805c2a0 0x7f1eb805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:12.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.918+0000 7f1ed0f0a700 1 --2- 172.21.15.139:0/3564513379 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1eb805c2a0 0x7f1eb805e760 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f1ebc009fd0 tx=0x7f1ebc00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:12.911 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:12.922+0000 7f1ec1ffb700 1 -- 172.21.15.139:0/3564513379 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ec804a220 con 0x7f1ecc0f55a0 2021-11-20T06:08:13.092 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:13.108+0000 7f1ed396f700 1 -- 172.21.15.139:0/3564513379 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ecc0fdd30 con 0x7f1ecc0f55a0 2021-11-20T06:08:13.093 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:13.108+0000 7f1ec1ffb700 1 -- 172.21.15.139:0/3564513379 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f1ec804d840 con 0x7f1ecc0f55a0 2021-11-20T06:08:13.093 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:13.094 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:02.313577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24549},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24549":{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:13.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:13.110+0000 7f1ed396f700 1 -- 172.21.15.139:0/3564513379 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1eb805c2a0 msgr2=0x7f1eb805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:13.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:13.110+0000 7f1ed396f700 1 --2- 172.21.15.139:0/3564513379 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1eb805c2a0 0x7f1eb805e760 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f1ebc009fd0 tx=0x7f1ebc00b040).stop 2021-11-20T06:08:13.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:13.110+0000 7f1ed396f700 1 -- 172.21.15.139:0/3564513379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ecc0f55a0 msgr2=0x7f1ecc105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:13.096 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:13.110+0000 7f1ed396f700 1 --2- 172.21.15.139:0/3564513379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ecc0f55a0 0x7f1ecc105eb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f1ec80078d0 tx=0x7f1ec8004d10).stop 2021-11-20T06:08:13.096 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:13.110+0000 7f1ed396f700 1 -- 172.21.15.139:0/3564513379 shutdown_connections 2021-11-20T06:08:13.096 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:13.110+0000 7f1ed396f700 1 --2- 172.21.15.139:0/3564513379 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1eb805c2a0 0x7f1eb805e760 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:13.096 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:13.110+0000 7f1ed396f700 1 --2- 172.21.15.139:0/3564513379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1ecc0f55a0 0x7f1ecc105eb0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:13.097 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:13.110+0000 7f1ed396f700 1 --2- 172.21.15.139:0/3564513379 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ecc0f6840 0x7f1ecc1063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:13.097 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:13.110+0000 7f1ed396f700 1 -- 172.21.15.139:0/3564513379 >> 172.21.15.139:0/3564513379 conn(0x7f1ecc0f0ce0 msgr2=0x7f1ecc0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:13.097 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:13.110+0000 7f1ed396f700 1 -- 172.21.15.139:0/3564513379 shutdown_connections 2021-11-20T06:08:13.097 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:13.110+0000 7f1ed396f700 1 -- 172.21.15.139:0/3564513379 wait complete. 2021-11-20T06:08:13.097 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 87 2021-11-20T06:08:13.106 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:02.313577+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24549}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24549': {'gid': 24549, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 84, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6827/2304740608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2304740608}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2304740608}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:13.107 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24549:up:rejoin 2021-11-20T06:08:13.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:13 smithi139 conmon[33501]: cluster 2021-11-20T06:08:12.281299+0000 2021-11-20T06:08:13.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:13 smithi139 conmon[33501]: mgr.smithi139.dplyxn (mgr.14162) 332 : cluster [DBG] pgmap v309: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:08:13.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:13 smithi139 conmon[33501]: audit 2021-11-20T06:08:13.109316+0000 mon.smithi139 (mon.0) 884 : audit [DBG] from='client.? 172.21.15.139:0/3564513379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:13.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:13 smithi178 conmon[36600]: cluster 2021-11-20T06:08:12.281299+0000 mgr.smithi139.dplyxn (mgr.14162) 2021-11-20T06:08:13.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:13 smithi178 conmon[36600]: 332 : cluster [DBG] pgmap v309: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-20T06:08:13.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:13 smithi178 conmon[36600]: audit 2021-11-20T06:08:13.109316+0000 mon.smithi139 (mon.0) 884 : audit [DBG] from='client.? 172.21.15.139:0/3564513379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:14.108 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:14.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.254+0000 7fd006b9a700 1 -- 172.21.15.139:0/1080379934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0000f6680 msgr2=0x7fd0000f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:14.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.254+0000 7fd006b9a700 1 --2- 172.21.15.139:0/1080379934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0000f6680 0x7fd0000f6aa0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7fcff4004660 tx=0x7fcff4009b30).stop 2021-11-20T06:08:14.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.254+0000 7fd006b9a700 1 -- 172.21.15.139:0/1080379934 shutdown_connections 2021-11-20T06:08:14.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.254+0000 7fd006b9a700 1 --2- 172.21.15.139:0/1080379934 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0000f7920 0x7fd0000f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:14.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.254+0000 7fd006b9a700 1 --2- 172.21.15.139:0/1080379934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0000f6680 0x7fd0000f6aa0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:14.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.254+0000 7fd006b9a700 1 -- 172.21.15.139:0/1080379934 >> 172.21.15.139:0/1080379934 conn(0x7fd0000f1de0 msgr2=0x7fd0000f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:14.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.254+0000 7fd006b9a700 1 -- 172.21.15.139:0/1080379934 shutdown_connections 2021-11-20T06:08:14.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.254+0000 7fd006b9a700 1 -- 172.21.15.139:0/1080379934 wait complete. 2021-11-20T06:08:14.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.255+0000 7fd006b9a700 1 Processor -- start 2021-11-20T06:08:14.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.255+0000 7fd006b9a700 1 -- start start 2021-11-20T06:08:14.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.255+0000 7fd006b9a700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0000f6680 0x7fd000106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:14.246 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.255+0000 7fd006b9a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0000f7920 0x7fd0001073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:14.246 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.255+0000 7fd006b9a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd000107950 con 0x7fd0000f7920 2021-11-20T06:08:14.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.255+0000 7fd006b9a700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd000107a90 con 0x7fd0000f6680 2021-11-20T06:08:14.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.255+0000 7fcffffff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0000f7920 0x7fd0001073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:14.247 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.255+0000 7fcffffff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0000f7920 0x7fd0001073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59158/0 (socket says 172.21.15.139:59158) 2021-11-20T06:08:14.248 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.255+0000 7fcffffff700 1 -- 172.21.15.139:0/3365247562 learned_addr learned my addr 172.21.15.139:0/3365247562 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:14.249 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.255+0000 7fd004936700 1 --2- 172.21.15.139:0/3365247562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0000f6680 0x7fd000106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:14.249 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.255+0000 7fcffffff700 1 -- 172.21.15.139:0/3365247562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0000f6680 msgr2=0x7fd000106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:14.249 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.255+0000 7fcffffff700 1 --2- 172.21.15.139:0/3365247562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0000f6680 0x7fd000106eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:14.250 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.255+0000 7fcffffff700 1 -- 172.21.15.139:0/3365247562 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff4005040 con 0x7fd0000f7920 2021-11-20T06:08:14.250 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.255+0000 7fcffffff700 1 --2- 172.21.15.139:0/3365247562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0000f7920 0x7fd0001073a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7fcfec00c790 tx=0x7fcfec007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:14.250 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.256+0000 7fcffd7fa700 1 -- 172.21.15.139:0/3365247562 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfec002590 con 0x7fd0000f7920 2021-11-20T06:08:14.250 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.256+0000 7fcffd7fa700 1 -- 172.21.15.139:0/3365247562 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfec002ba0 con 0x7fd0000f7920 2021-11-20T06:08:14.251 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.256+0000 7fd006b9a700 1 -- 172.21.15.139:0/3365247562 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00010c530 con 0x7fd0000f7920 2021-11-20T06:08:14.251 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.256+0000 7fcffd7fa700 1 -- 172.21.15.139:0/3365247562 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfec013d90 con 0x7fd0000f7920 2021-11-20T06:08:14.251 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.256+0000 7fd006b9a700 1 -- 172.21.15.139:0/3365247562 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00010caf0 con 0x7fd0000f7920 2021-11-20T06:08:14.251 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.256+0000 7fd006b9a700 1 -- 172.21.15.139:0/3365247562 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd000027a00 con 0x7fd0000f7920 2021-11-20T06:08:14.251 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.257+0000 7fcffd7fa700 1 -- 172.21.15.139:0/3365247562 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcfec01a060 con 0x7fd0000f7920 2021-11-20T06:08:14.252 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.257+0000 7fcffd7fa700 1 --2- 172.21.15.139:0/3365247562 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fcff005c2a0 0x7fcff005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:14.252 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.257+0000 7fcffd7fa700 1 -- 172.21.15.139:0/3365247562 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fcfec076620 con 0x7fd0000f7920 2021-11-20T06:08:14.252 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.257+0000 7fd004936700 1 --2- 172.21.15.139:0/3365247562 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fcff005c2a0 0x7fcff005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:14.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.258+0000 7fd004936700 1 --2- 172.21.15.139:0/3365247562 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fcff005c2a0 0x7fcff005e760 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fcff4004630 tx=0x7fcff4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:14.253 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.261+0000 7fcffd7fa700 1 -- 172.21.15.139:0/3365247562 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfec048e00 con 0x7fd0000f7920 2021-11-20T06:08:14.431 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.447+0000 7fd006b9a700 1 -- 172.21.15.139:0/3365247562 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0000fee10 con 0x7fd0000f7920 2021-11-20T06:08:14.432 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.447+0000 7fcffd7fa700 1 -- 172.21.15.139:0/3365247562 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2187 (secure 0 0 0) 0x7fcfec04c420 con 0x7fd0000f7920 2021-11-20T06:08:14.433 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:14.433 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:02.313577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24549},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24549":{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:14.434 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.449+0000 7fd006b9a700 1 -- 172.21.15.139:0/3365247562 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fcff005c2a0 msgr2=0x7fcff005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:14.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.449+0000 7fd006b9a700 1 --2- 172.21.15.139:0/3365247562 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fcff005c2a0 0x7fcff005e760 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fcff4004630 tx=0x7fcff4009b30).stop 2021-11-20T06:08:14.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.449+0000 7fd006b9a700 1 -- 172.21.15.139:0/3365247562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0000f7920 msgr2=0x7fd0001073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:14.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.449+0000 7fd006b9a700 1 --2- 172.21.15.139:0/3365247562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0000f7920 0x7fd0001073a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7fcfec00c790 tx=0x7fcfec007d70).stop 2021-11-20T06:08:14.435 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.449+0000 7fd006b9a700 1 -- 172.21.15.139:0/3365247562 shutdown_connections 2021-11-20T06:08:14.436 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.449+0000 7fd006b9a700 1 --2- 172.21.15.139:0/3365247562 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fcff005c2a0 0x7fcff005e760 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:14.436 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.449+0000 7fd006b9a700 1 --2- 172.21.15.139:0/3365247562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0000f6680 0x7fd000106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:14.436 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.449+0000 7fd006b9a700 1 --2- 172.21.15.139:0/3365247562 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd0000f7920 0x7fd0001073a0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:14.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.449+0000 7fd006b9a700 1 -- 172.21.15.139:0/3365247562 >> 172.21.15.139:0/3365247562 conn(0x7fd0000f1de0 msgr2=0x7fd0000fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:14.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.449+0000 7fd006b9a700 1 -- 172.21.15.139:0/3365247562 shutdown_connections 2021-11-20T06:08:14.437 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:14.449+0000 7fd006b9a700 1 -- 172.21.15.139:0/3365247562 wait complete. 2021-11-20T06:08:14.437 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 87 2021-11-20T06:08:14.446 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:02.313577+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24549}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24549': {'gid': 24549, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 84, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6827/2304740608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2304740608}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2304740608}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:14.446 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24549:up:rejoin 2021-11-20T06:08:15.448 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:15.576 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.587+0000 7f53960f8700 1 -- 172.21.15.139:0/4270318261 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53900f7940 msgr2=0x7f53900f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:15.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.587+0000 7f53960f8700 1 --2- 172.21.15.139:0/4270318261 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53900f7940 0x7f53900f7da0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f5378004660 tx=0x7f5378009b30).stop 2021-11-20T06:08:15.577 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.587+0000 7f53960f8700 1 -- 172.21.15.139:0/4270318261 shutdown_connections 2021-11-20T06:08:15.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.587+0000 7f53960f8700 1 --2- 172.21.15.139:0/4270318261 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53900f7940 0x7f53900f7da0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:15.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.587+0000 7f53960f8700 1 --2- 172.21.15.139:0/4270318261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53900f66a0 0x7f53900f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:15.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.587+0000 7f53960f8700 1 -- 172.21.15.139:0/4270318261 >> 172.21.15.139:0/4270318261 conn(0x7f53900f1da0 msgr2=0x7f53900f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:15.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.587+0000 7f53960f8700 1 -- 172.21.15.139:0/4270318261 shutdown_connections 2021-11-20T06:08:15.578 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.587+0000 7f53960f8700 1 -- 172.21.15.139:0/4270318261 wait complete. 2021-11-20T06:08:15.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.587+0000 7f53960f8700 1 Processor -- start 2021-11-20T06:08:15.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.587+0000 7f53960f8700 1 -- start start 2021-11-20T06:08:15.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f53960f8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53900f66a0 0x7f5390106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:15.579 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f53960f8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53900f7940 0x7f53901073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:15.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f53960f8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5390107960 con 0x7f53900f7940 2021-11-20T06:08:15.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f53960f8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5390107aa0 con 0x7f53900f66a0 2021-11-20T06:08:15.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f5386ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53900f7940 0x7f53901073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:15.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f5386ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53900f7940 0x7f53901073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59168/0 (socket says 172.21.15.139:59168) 2021-11-20T06:08:15.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f5386ffd700 1 -- 172.21.15.139:0/2981896677 learned_addr learned my addr 172.21.15.139:0/2981896677 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:15.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f538f7fe700 1 --2- 172.21.15.139:0/2981896677 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53900f66a0 0x7f5390106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:15.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f5386ffd700 1 -- 172.21.15.139:0/2981896677 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53900f66a0 msgr2=0x7f5390106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:15.582 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f5386ffd700 1 --2- 172.21.15.139:0/2981896677 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53900f66a0 0x7f5390106ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:15.582 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f5386ffd700 1 -- 172.21.15.139:0/2981896677 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5378005040 con 0x7f53900f7940 2021-11-20T06:08:15.582 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f5386ffd700 1 --2- 172.21.15.139:0/2981896677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53900f7940 0x7f53901073b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f5378009f50 tx=0x7f5378009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:15.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f538cff9700 1 -- 172.21.15.139:0/2981896677 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5378018070 con 0x7f53900f7940 2021-11-20T06:08:15.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f538cff9700 1 -- 172.21.15.139:0/2981896677 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5378013410 con 0x7f53900f7940 2021-11-20T06:08:15.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f53960f8700 1 -- 172.21.15.139:0/2981896677 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f539010c4e0 con 0x7f53900f7940 2021-11-20T06:08:15.584 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f538cff9700 1 -- 172.21.15.139:0/2981896677 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f537801c3c0 con 0x7f53900f7940 2021-11-20T06:08:15.584 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.588+0000 7f53960f8700 1 -- 172.21.15.139:0/2981896677 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f539010caa0 con 0x7f53900f7940 2021-11-20T06:08:15.584 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.589+0000 7f53960f8700 1 -- 172.21.15.139:0/2981896677 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5390027a00 con 0x7f53900f7940 2021-11-20T06:08:15.585 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.589+0000 7f538cff9700 1 -- 172.21.15.139:0/2981896677 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f537801d420 con 0x7f53900f7940 2021-11-20T06:08:15.585 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.590+0000 7f538cff9700 1 --2- 172.21.15.139:0/2981896677 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f537c05c2a0 0x7f537c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:15.585 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.590+0000 7f538cff9700 1 -- 172.21.15.139:0/2981896677 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f53780780a0 con 0x7f53900f7940 2021-11-20T06:08:15.586 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.590+0000 7f538f7fe700 1 --2- 172.21.15.139:0/2981896677 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f537c05c2a0 0x7f537c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:15.586 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.590+0000 7f538f7fe700 1 --2- 172.21.15.139:0/2981896677 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f537c05c2a0 0x7f537c05e760 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f53900f77a0 tx=0x7f538000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:15.587 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.594+0000 7f538cff9700 1 -- 172.21.15.139:0/2981896677 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f537804a800 con 0x7f53900f7940 2021-11-20T06:08:15.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:15 smithi178 conmon[36600]: cluster 2021-11-20T06:08:14.281706+0000 mgr.smithi139.dplyxn (mgr.14162) 333 : cluster [DBG] 2021-11-20T06:08:15.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:15 smithi178 conmon[36600]: pgmap v310: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2021-11-20T06:08:15.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:15 smithi178 conmon[36600]: audit 2021-11-20T06:08:14.448310+0000 mon.smithi139 (mon.0) 885 : audit [DBG] from='client.? 172.21.15.139:0/3365247562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:15.761 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.777+0000 7f53960f8700 1 -- 172.21.15.139:0/2981896677 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53900fee30 con 0x7f53900f7940 2021-11-20T06:08:15.762 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.777+0000 7f538cff9700 1 -- 172.21.15.139:0/2981896677 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f537804de20 con 0x7f53900f7940 2021-11-20T06:08:15.762 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:15.763 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:02.313577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24549},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24549":{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:15.764 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.779+0000 7f53960f8700 1 -- 172.21.15.139:0/2981896677 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f537c05c2a0 msgr2=0x7f537c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:15.765 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.779+0000 7f53960f8700 1 --2- 172.21.15.139:0/2981896677 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f537c05c2a0 0x7f537c05e760 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f53900f77a0 tx=0x7f538000b040).stop 2021-11-20T06:08:15.765 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.779+0000 7f53960f8700 1 -- 172.21.15.139:0/2981896677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53900f7940 msgr2=0x7f53901073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:15.765 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.779+0000 7f53960f8700 1 --2- 172.21.15.139:0/2981896677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53900f7940 0x7f53901073b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f5378009f50 tx=0x7f5378009b30).stop 2021-11-20T06:08:15.765 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.779+0000 7f53960f8700 1 -- 172.21.15.139:0/2981896677 shutdown_connections 2021-11-20T06:08:15.766 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.779+0000 7f53960f8700 1 --2- 172.21.15.139:0/2981896677 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f537c05c2a0 0x7f537c05e760 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:15.766 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.779+0000 7f53960f8700 1 --2- 172.21.15.139:0/2981896677 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53900f66a0 0x7f5390106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:15.766 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.779+0000 7f53960f8700 1 --2- 172.21.15.139:0/2981896677 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53900f7940 0x7f53901073b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:15.766 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.779+0000 7f53960f8700 1 -- 172.21.15.139:0/2981896677 >> 172.21.15.139:0/2981896677 conn(0x7f53900f1da0 msgr2=0x7f53900fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:15.767 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.779+0000 7f53960f8700 1 -- 172.21.15.139:0/2981896677 shutdown_connections 2021-11-20T06:08:15.767 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:15.779+0000 7f53960f8700 1 -- 172.21.15.139:0/2981896677 wait complete. 2021-11-20T06:08:15.767 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 87 2021-11-20T06:08:15.774 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:15 smithi139 conmon[33501]: cluster 2021-11-20T06:08:14.281706+0000 mgr.smithi139.dplyxn (mgr.14162) 333 : cluster [DBG] 2021-11-20T06:08:15.775 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:15 smithi139 conmon[33501]: pgmap v310: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2021-11-20T06:08:15.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:15 smithi139 conmon[33501]: audit 2021-11-20T06:08:14.448310+0000 mon.smithi139 (mon.0) 885 : audit [DBG] from='client.? 172.21.15.139:0/3365247562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:15.777 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:02.313577+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24549}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24549': {'gid': 24549, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 84, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6827/2304740608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2304740608}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2304740608}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:15.777 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24549:up:rejoin 2021-11-20T06:08:16.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:16 smithi178 conmon[36600]: audit 2021-11-20T06:08:15.778217+0000 mon.smithi139 (mon.0) 886 : 2021-11-20T06:08:16.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:16 smithi178 conmon[36600]: audit [DBG] from='client.? 172.21.15.139:0/2981896677' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:16.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:16 smithi139 conmon[33501]: audit 2021-11-20T06:08:15.778217+0000 mon.smithi139 (mon.0) 886 : audit [DBG] from='client.? 172.21.15.139:0/2981896677' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:16.778 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:16.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.920+0000 7f9b94fba700 1 -- 172.21.15.139:0/2041143321 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b900f6850 msgr2=0x7f9b900f6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:16.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.920+0000 7f9b94fba700 1 --2- 172.21.15.139:0/2041143321 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b900f6850 0x7f9b900f6cb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f9b84004660 tx=0x7f9b84009b30).stop 2021-11-20T06:08:16.907 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.920+0000 7f9b94fba700 1 -- 172.21.15.139:0/2041143321 shutdown_connections 2021-11-20T06:08:16.907 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.920+0000 7f9b94fba700 1 --2- 172.21.15.139:0/2041143321 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b900f6850 0x7f9b900f6cb0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:16.907 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.920+0000 7f9b94fba700 1 --2- 172.21.15.139:0/2041143321 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b900f55b0 0x7f9b900f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:16.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.920+0000 7f9b94fba700 1 -- 172.21.15.139:0/2041143321 >> 172.21.15.139:0/2041143321 conn(0x7f9b900f0d10 msgr2=0x7f9b900f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:16.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.920+0000 7f9b94fba700 1 -- 172.21.15.139:0/2041143321 shutdown_connections 2021-11-20T06:08:16.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.920+0000 7f9b94fba700 1 -- 172.21.15.139:0/2041143321 wait complete. 2021-11-20T06:08:16.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.920+0000 7f9b94fba700 1 Processor -- start 2021-11-20T06:08:16.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.920+0000 7f9b94fba700 1 -- start start 2021-11-20T06:08:16.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.920+0000 7f9b94fba700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b900f55b0 0x7f9b90106e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:16.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.920+0000 7f9b94fba700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b900f6850 0x7f9b90107370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:16.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.920+0000 7f9b94fba700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b90107920 con 0x7f9b900f6850 2021-11-20T06:08:16.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.920+0000 7f9b94fba700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b90107a60 con 0x7f9b900f55b0 2021-11-20T06:08:16.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.921+0000 7f9b8dd9b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b900f6850 0x7f9b90107370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:16.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.921+0000 7f9b8dd9b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b900f6850 0x7f9b90107370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59178/0 (socket says 172.21.15.139:59178) 2021-11-20T06:08:16.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.921+0000 7f9b8dd9b700 1 -- 172.21.15.139:0/3354583934 learned_addr learned my addr 172.21.15.139:0/3354583934 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:16.911 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.921+0000 7f9b8e59c700 1 --2- 172.21.15.139:0/3354583934 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b900f55b0 0x7f9b90106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:16.911 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.921+0000 7f9b8dd9b700 1 -- 172.21.15.139:0/3354583934 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b900f55b0 msgr2=0x7f9b90106e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:16.911 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.921+0000 7f9b8dd9b700 1 --2- 172.21.15.139:0/3354583934 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b900f55b0 0x7f9b90106e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:16.911 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.921+0000 7f9b8dd9b700 1 -- 172.21.15.139:0/3354583934 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b84005040 con 0x7f9b900f6850 2021-11-20T06:08:16.912 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.921+0000 7f9b8dd9b700 1 --2- 172.21.15.139:0/3354583934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b900f6850 0x7f9b90107370 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f9b84009f50 tx=0x7f9b84009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:16.912 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.921+0000 7f9b7effd700 1 -- 172.21.15.139:0/3354583934 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b84018070 con 0x7f9b900f6850 2021-11-20T06:08:16.912 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.921+0000 7f9b7effd700 1 -- 172.21.15.139:0/3354583934 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b84013410 con 0x7f9b900f6850 2021-11-20T06:08:16.912 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.921+0000 7f9b94fba700 1 -- 172.21.15.139:0/3354583934 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9010c4a0 con 0x7f9b900f6850 2021-11-20T06:08:16.912 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.921+0000 7f9b7effd700 1 -- 172.21.15.139:0/3354583934 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b8401c3c0 con 0x7f9b900f6850 2021-11-20T06:08:16.913 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.921+0000 7f9b94fba700 1 -- 172.21.15.139:0/3354583934 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9010ca60 con 0x7f9b900f6850 2021-11-20T06:08:16.913 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.922+0000 7f9b94fba700 1 -- 172.21.15.139:0/3354583934 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b90027a00 con 0x7f9b900f6850 2021-11-20T06:08:16.914 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.922+0000 7f9b7effd700 1 -- 172.21.15.139:0/3354583934 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9b8401d420 con 0x7f9b900f6850 2021-11-20T06:08:16.914 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.923+0000 7f9b7effd700 1 --2- 172.21.15.139:0/3354583934 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9b7805c2a0 0x7f9b7805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:16.915 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.923+0000 7f9b7effd700 1 -- 172.21.15.139:0/3354583934 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f9b84077ff0 con 0x7f9b900f6850 2021-11-20T06:08:16.915 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.923+0000 7f9b8e59c700 1 --2- 172.21.15.139:0/3354583934 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9b7805c2a0 0x7f9b7805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:16.915 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.923+0000 7f9b8e59c700 1 --2- 172.21.15.139:0/3354583934 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9b7805c2a0 0x7f9b7805e760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f9b900f66b0 tx=0x7f9b8000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:16.915 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:16.927+0000 7f9b7effd700 1 -- 172.21.15.139:0/3354583934 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b8404a7d0 con 0x7f9b900f6850 2021-11-20T06:08:17.094 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:17.110+0000 7f9b94fba700 1 -- 172.21.15.139:0/3354583934 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b900fee40 con 0x7f9b900f6850 2021-11-20T06:08:17.095 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:17.110+0000 7f9b7effd700 1 -- 172.21.15.139:0/3354583934 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2187 (secure 0 0 0) 0x7f9b8404ddf0 con 0x7f9b900f6850 2021-11-20T06:08:17.095 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:17.096 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:02.313577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24549},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24549":{"gid":24549,"name":"cephfs.smithi178.pskihy","rank":0,"incarnation":84,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6827/2304740608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2304740608},{"type":"v1","addr":"172.21.15.178:6827","nonce":2304740608}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:17.097 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:17.112+0000 7f9b94fba700 1 -- 172.21.15.139:0/3354583934 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9b7805c2a0 msgr2=0x7f9b7805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:17.097 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:17.112+0000 7f9b94fba700 1 --2- 172.21.15.139:0/3354583934 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9b7805c2a0 0x7f9b7805e760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f9b900f66b0 tx=0x7f9b8000b040).stop 2021-11-20T06:08:17.098 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:17.112+0000 7f9b94fba700 1 -- 172.21.15.139:0/3354583934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b900f6850 msgr2=0x7f9b90107370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:17.098 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:17.112+0000 7f9b94fba700 1 --2- 172.21.15.139:0/3354583934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b900f6850 0x7f9b90107370 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f9b84009f50 tx=0x7f9b84009b30).stop 2021-11-20T06:08:17.099 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:17.112+0000 7f9b94fba700 1 -- 172.21.15.139:0/3354583934 shutdown_connections 2021-11-20T06:08:17.099 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:17.112+0000 7f9b94fba700 1 --2- 172.21.15.139:0/3354583934 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f9b7805c2a0 0x7f9b7805e760 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:17.099 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:17.112+0000 7f9b94fba700 1 --2- 172.21.15.139:0/3354583934 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b900f55b0 0x7f9b90106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:17.099 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:17.112+0000 7f9b94fba700 1 --2- 172.21.15.139:0/3354583934 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9b900f6850 0x7f9b90107370 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:17.100 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:17.112+0000 7f9b94fba700 1 -- 172.21.15.139:0/3354583934 >> 172.21.15.139:0/3354583934 conn(0x7f9b900f0d10 msgr2=0x7f9b900fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:17.100 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:17.112+0000 7f9b94fba700 1 -- 172.21.15.139:0/3354583934 shutdown_connections 2021-11-20T06:08:17.100 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:17.112+0000 7f9b94fba700 1 -- 172.21.15.139:0/3354583934 wait complete. 2021-11-20T06:08:17.100 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 87 2021-11-20T06:08:17.108 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:02.313577+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24549}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24549': {'gid': 24549, 'name': 'cephfs.smithi178.pskihy', 'rank': 0, 'incarnation': 84, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6827/2304740608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6826', 'nonce': 2304740608}, {'type': 'v1', 'addr': '172.21.15.178:6827', 'nonce': 2304740608}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:17.109 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24549:up:rejoin 2021-11-20T06:08:17.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: cluster 2021-11-20T06:08:16.282099+0000 mgr.smithi139.dplyxn (mgr.14162) 334 : cluster 2021-11-20T06:08:17.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: [DBG] pgmap v311: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-20T06:08:17.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: audit 2021-11-20T06:08:17.111276+0000 mon.smithi139 (mon.0) 887 : audit [DBG] from='client.? 172.21.15.139:0/3354583934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:17.528 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: cluster 2021-11-20T06:08:17.235131+0000 mon.smithi139 (mon. 2021-11-20T06:08:17.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: 0) 888 : cluster [INF] daemon mds.cephfs.smithi178.pskihy restarted 2021-11-20T06:08:17.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: cluster 2021-11-20T06:08:17.238609+0000 mon.smithi139 (mon.0) 889 : cluster 2021-11-20T06:08:17.530 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: [DBG] osdmap e58: 6 total, 6 up, 6 in 2021-11-20T06:08:17.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: cluster 2021-11-20T06 2021-11-20T06:08:17.531 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: :08:17.249499+0000 mon.smithi139 (mon.0) 890 : cluster [INF] Standby daemon mds.cephfs.smithi178.blvkbx assigned to filesystem cephfs as rank 0 2021-11-20T06:08:17.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: cluster 2021-11-20T06:08:17.252190 2021-11-20T06:08:17.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: +0000 mon.smithi139 (mon.0) 891 : cluster [DBG] mds.? [v2:172.21.15.178:6826/339439062,v1:172.21.15.178:6827/339439062] up:boot 2021-11-20T06:08:17.532 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: cluster 2021-11-20T06: 2021-11-20T06:08:17.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: 08:17.252232+0000 mon.smithi139 (mon.0) 892 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:replay} 1 up:standby 2021-11-20T06:08:17.533 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: 2021-11-20T06:08:17.534 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: audit 2021- 2021-11-20T06:08:17.534 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: 11-20T06:08:17.252467+0000 mon.smithi139 (mon.0) 893 : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.pskihy"}]: dispatch 2021-11-20T06:08:17.535 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:17 smithi139 conmon[33501]: 2021-11-20T06:08:17.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: cluster 2021-11-20T06:08:16.282099+0000 mgr.smithi139.dplyxn (mgr.14162) 334 : cluster [DBG] pgmap v311: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-20T06:08:17.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: audit 2021-11-20T06:08:17.111276+0000 mon.smithi139 (mon 2021-11-20T06:08:17.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: .0) 887 : audit [DBG] from='client.? 172.21.15.139:0/3354583934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:17.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: cluster 2021-11-20T06:08:17.235131+0000 mon.smithi139 (mon.0) 888 : cluster [INF] daemon mds.cephfs.smithi178.pskihy restarted 2021-11-20T06:08:17.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: cluster 2021-11-20T06:08:17.238609+0000 mon.smithi139 (mon.0) 889 : cluster [DBG] osdmap e58: 6 total, 6 up, 6 in 2021-11-20T06:08:17.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: cluster 2021 2021-11-20T06:08:17.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: -11-20T06:08:17.249499+0000 mon.smithi139 (mon.0) 890 : cluster [INF] Standby daemon mds.cephfs.smithi178.blvkbx assigned to filesystem cephfs as rank 0 2021-11-20T06:08:17.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: cluster 2021- 2021-11-20T06:08:17.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: 11-20T06:08:17.252190+0000 mon.smithi139 (mon.0) 891 2021-11-20T06:08:17.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: : cluster [DBG] mds.? [v2:172.21.15.178:6826/339439062,v1:172.21.15.178:6827/339439062] up:boot 2021-11-20T06:08:17.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: cluster 2021-11-20T06: 2021-11-20T06:08:17.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: 08:17.252232+0000 mon.smithi139 (mon.0) 892 2021-11-20T06:08:17.694 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:replay} 1 up:standby 2021-11-20T06:08:17.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: audit 2021-11-20T06: 2021-11-20T06:08:17.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: 08:17.252467+0000 mon.smithi139 (mon.0) 893 2021-11-20T06:08:17.695 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:17 smithi178 conmon[36600]: : audit [DBG] from='mgr.14162 172.21.15.139:0/4031711332' entity='mgr.smithi139.dplyxn' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi178.pskihy"}]: dispatch 2021-11-20T06:08:18.110 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:18.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.248+0000 7f1bb9379700 1 -- 172.21.15.139:0/3870098391 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bb40f65a0 msgr2=0x7f1bb40f69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:18.236 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.248+0000 7f1bb9379700 1 --2- 172.21.15.139:0/3870098391 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bb40f65a0 0x7f1bb40f69c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f1ba4004660 tx=0x7f1ba4009b30).stop 2021-11-20T06:08:18.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.248+0000 7f1bb9379700 1 -- 172.21.15.139:0/3870098391 shutdown_connections 2021-11-20T06:08:18.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.248+0000 7f1bb9379700 1 --2- 172.21.15.139:0/3870098391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1bb40f7840 0x7f1bb40f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:18.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.248+0000 7f1bb9379700 1 --2- 172.21.15.139:0/3870098391 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bb40f65a0 0x7f1bb40f69c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:18.237 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.248+0000 7f1bb9379700 1 -- 172.21.15.139:0/3870098391 >> 172.21.15.139:0/3870098391 conn(0x7f1bb40f1ce0 msgr2=0x7f1bb40f4120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:18.238 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.249+0000 7f1bb9379700 1 -- 172.21.15.139:0/3870098391 shutdown_connections 2021-11-20T06:08:18.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.249+0000 7f1bb9379700 1 -- 172.21.15.139:0/3870098391 wait complete. 2021-11-20T06:08:18.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.249+0000 7f1bb9379700 1 Processor -- start 2021-11-20T06:08:18.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.249+0000 7f1bb9379700 1 -- start start 2021-11-20T06:08:18.239 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.249+0000 7f1bb9379700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1bb40f65a0 0x7f1bb4106d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:18.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.249+0000 7f1bb9379700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bb40f7840 0x7f1bb4107260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:18.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.249+0000 7f1bb9379700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb4107860 con 0x7f1bb40f7840 2021-11-20T06:08:18.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.249+0000 7f1bb9379700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb4108070 con 0x7f1bb40f65a0 2021-11-20T06:08:18.240 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.249+0000 7f1bb37fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bb40f7840 0x7f1bb4107260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:18.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.249+0000 7f1bb37fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bb40f7840 0x7f1bb4107260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59186/0 (socket says 172.21.15.139:59186) 2021-11-20T06:08:18.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.249+0000 7f1bb37fe700 1 -- 172.21.15.139:0/2987619450 learned_addr learned my addr 172.21.15.139:0/2987619450 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:18.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.250+0000 7f1bb3fff700 1 --2- 172.21.15.139:0/2987619450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1bb40f65a0 0x7f1bb4106d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:18.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.250+0000 7f1bb37fe700 1 -- 172.21.15.139:0/2987619450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1bb40f65a0 msgr2=0x7f1bb4106d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:18.241 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.250+0000 7f1bb37fe700 1 --2- 172.21.15.139:0/2987619450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1bb40f65a0 0x7f1bb4106d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:18.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.250+0000 7f1bb37fe700 1 -- 172.21.15.139:0/2987619450 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba4005040 con 0x7f1bb40f7840 2021-11-20T06:08:18.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.250+0000 7f1bb37fe700 1 --2- 172.21.15.139:0/2987619450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bb40f7840 0x7f1bb4107260 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f1ba8002700 tx=0x7f1ba8009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:18.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.250+0000 7f1bb0ff9700 1 -- 172.21.15.139:0/2987619450 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ba8009320 con 0x7f1bb40f7840 2021-11-20T06:08:18.242 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.250+0000 7f1bb0ff9700 1 -- 172.21.15.139:0/2987619450 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ba800f040 con 0x7f1bb40f7840 2021-11-20T06:08:18.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.250+0000 7f1bb9379700 1 -- 172.21.15.139:0/2987619450 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb41082f0 con 0x7f1bb40f7840 2021-11-20T06:08:18.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.250+0000 7f1bb0ff9700 1 -- 172.21.15.139:0/2987619450 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ba8014870 con 0x7f1bb40f7840 2021-11-20T06:08:18.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.250+0000 7f1bb9379700 1 -- 172.21.15.139:0/2987619450 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb41088b0 con 0x7f1bb40f7840 2021-11-20T06:08:18.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.251+0000 7f1bb9379700 1 -- 172.21.15.139:0/2987619450 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb40f5990 con 0x7f1bb40f7840 2021-11-20T06:08:18.243 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.251+0000 7f1bb0ff9700 1 -- 172.21.15.139:0/2987619450 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1ba8009480 con 0x7f1bb40f7840 2021-11-20T06:08:18.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.252+0000 7f1bb0ff9700 1 --2- 172.21.15.139:0/2987619450 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1b9c05c300 0x7f1b9c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:18.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.252+0000 7f1bb0ff9700 1 -- 172.21.15.139:0/2987619450 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f1ba8076700 con 0x7f1bb40f7840 2021-11-20T06:08:18.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.252+0000 7f1bb3fff700 1 --2- 172.21.15.139:0/2987619450 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1b9c05c300 0x7f1b9c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:18.244 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.252+0000 7f1bb3fff700 1 --2- 172.21.15.139:0/2987619450 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1b9c05c300 0x7f1b9c05e7c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f1bb4106ae0 tx=0x7f1ba4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:18.245 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.255+0000 7f1bb0ff9700 1 -- 172.21.15.139:0/2987619450 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ba8048e10 con 0x7f1bb40f7840 2021-11-20T06:08:18.441 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.454+0000 7f1bb9379700 1 -- 172.21.15.139:0/2987619450 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bb40feec0 con 0x7f1bb40f7840 2021-11-20T06:08:18.442 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.455+0000 7f1bb0ff9700 1 -- 172.21.15.139:0/2987619450 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+2188 (secure 0 0 0) 0x7f1bb40feec0 con 0x7f1bb40f7840 2021-11-20T06:08:18.442 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:18.443 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24615,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/339439062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":339439062},{"type":"v1","addr":"172.21.15.178:6827","nonce":339439062}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":88}],"filesystems":[{"mdsmap":{"epoch":89,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:18.256448+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24583},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":88,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:18.444 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.456+0000 7f1bb9379700 1 -- 172.21.15.139:0/2987619450 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1b9c05c300 msgr2=0x7f1b9c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:18.445 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.456+0000 7f1bb9379700 1 --2- 172.21.15.139:0/2987619450 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1b9c05c300 0x7f1b9c05e7c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f1bb4106ae0 tx=0x7f1ba4009b30).stop 2021-11-20T06:08:18.445 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.456+0000 7f1bb9379700 1 -- 172.21.15.139:0/2987619450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bb40f7840 msgr2=0x7f1bb4107260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:18.445 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.456+0000 7f1bb9379700 1 --2- 172.21.15.139:0/2987619450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bb40f7840 0x7f1bb4107260 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f1ba8002700 tx=0x7f1ba8009e00).stop 2021-11-20T06:08:18.445 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.456+0000 7f1bb9379700 1 -- 172.21.15.139:0/2987619450 shutdown_connections 2021-11-20T06:08:18.446 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.456+0000 7f1bb9379700 1 --2- 172.21.15.139:0/2987619450 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f1b9c05c300 0x7f1b9c05e7c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:18.446 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.456+0000 7f1bb9379700 1 --2- 172.21.15.139:0/2987619450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1bb40f65a0 0x7f1bb4106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:18.446 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.456+0000 7f1bb9379700 1 --2- 172.21.15.139:0/2987619450 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1bb40f7840 0x7f1bb4107260 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:18.447 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.456+0000 7f1bb9379700 1 -- 172.21.15.139:0/2987619450 >> 172.21.15.139:0/2987619450 conn(0x7f1bb40f1ce0 msgr2=0x7f1bb40fb290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:18.447 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.456+0000 7f1bb9379700 1 -- 172.21.15.139:0/2987619450 shutdown_connections 2021-11-20T06:08:18.447 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:18.456+0000 7f1bb9379700 1 -- 172.21.15.139:0/2987619450 wait complete. 2021-11-20T06:08:18.447 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 89 2021-11-20T06:08:18.453 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:18.256448+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24583}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 88, 'state': 'up:reconnect', 'state_seq': 6, 'addr': '172.21.15.178:6825/4081910640', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 4081910640}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 4081910640}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:18.453 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:reconnect 2021-11-20T06:08:19.454 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:19.481 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:19 smithi139 conmon[33501]: cluster 2021-11-20T06:08:18.260382+0000 mon.smithi139 (mon.0) 894 : cluster 2021-11-20T06:08:19.482 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:19 smithi139 conmon[33501]: [DBG] mds.? [v2:172.21.15.178:6824/4081910640,v1:172.21.15.178:6825/4081910640] up:reconnect 2021-11-20T06:08:19.482 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:19 smithi139 conmon[33501]: cluster 2021-11-20T06:08:18.260457+0000 mon.smithi139 (mon.0) 895 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:reconnect} 1 up:standby 2021-11-20T06:08:19.482 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:19 smithi139 conmon[33501]: cluster 2021-11 2021-11-20T06:08:19.482 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:19 smithi139 conmon[33501]: -20T06:08:18.279995+0000 mds.cephfs.smithi178.blvkbx ( 2021-11-20T06:08:19.483 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:19 smithi139 conmon[33501]: mds.0) 1 : cluster [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0219996 2021-11-20T06:08:19.483 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:19 smithi139 conmon[33501]: cluster 2021-11-20T06:08:18. 2021-11-20T06:08:19.483 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:19 smithi139 conmon[33501]: 282485+0000 mgr.smithi139.dplyxn (mgr.14162) 335 : cluster [DBG] pgmap v313: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:08:19.483 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:19 smithi139 conmon[33501]: audit 2021-11-20T06:08:18.455417+0000 mon.smithi139 2021-11-20T06:08:19.484 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:19 smithi139 conmon[33501]: (mon.0) 896 : audit [DBG] from='client.? 172.21.15.139:0/2987619450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:19.563 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:19 smithi178 conmon[36600]: cluster 2021-11-20T06:08:18.260382+0000 mon.smithi139 (mon.0 2021-11-20T06:08:19.563 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:19 smithi178 conmon[36600]: ) 894 : cluster [DBG] mds.? [v2:172.21.15.178:6824/4081910640,v1:172.21.15.178:6825/4081910640] up:reconnect 2021-11-20T06:08:19.564 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:19 smithi178 conmon[36600]: cluster 2021-11-20T06:08:18.260457+0000 mon.smithi139 (mon.0) 895 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:reconnect} 1 up:standby 2021-11-20T06:08:19.564 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:19 smithi178 conmon[36600]: cluster 2021-11-20T06:08:19.564 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:19 smithi178 conmon[36600]: 2021-11-20T06:08:18.279995+0000 mds.cephfs.smithi178.blvkbx (mds.0) 1 : cluster [DBG] reconnect by client.24303 v1:172.21.15.139:0/938530995 after 0.0219996 2021-11-20T06:08:19.564 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:19 smithi178 conmon[36600]: cluster 2021-11-20T06: 2021-11-20T06:08:19.565 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:19 smithi178 conmon[36600]: 08:18.282485+0000 mgr.smithi139.dplyxn (mgr.14162) 335 2021-11-20T06:08:19.565 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:19 smithi178 conmon[36600]: : cluster [DBG] pgmap v313: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-20T06:08:19.565 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:19 smithi178 conmon[36600]: audit 2021-11-20T06 2021-11-20T06:08:19.565 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:19 smithi178 conmon[36600]: :08:18.455417+0000 mon.smithi139 (mon.0) 2021-11-20T06:08:19.565 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:19 smithi178 conmon[36600]: 896 : audit [DBG] from='client.? 172.21.15.139:0/2987619450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:19.580 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.594+0000 7f8bc2567700 1 -- 172.21.15.139:0/145519442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8bbc0f5b70 msgr2=0x7f8bbc0f5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:19.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.594+0000 7f8bc2567700 1 --2- 172.21.15.139:0/145519442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8bbc0f5b70 0x7f8bbc0f5f90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f8bac004660 tx=0x7f8bac009b30).stop 2021-11-20T06:08:19.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.594+0000 7f8bc2567700 1 -- 172.21.15.139:0/145519442 shutdown_connections 2021-11-20T06:08:19.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.594+0000 7f8bc2567700 1 --2- 172.21.15.139:0/145519442 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bbc0f6510 0x7f8bbc0f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:19.581 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.594+0000 7f8bc2567700 1 --2- 172.21.15.139:0/145519442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8bbc0f5b70 0x7f8bbc0f5f90 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:19.582 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.594+0000 7f8bc2567700 1 -- 172.21.15.139:0/145519442 >> 172.21.15.139:0/145519442 conn(0x7f8bbc0f0440 msgr2=0x7f8bbc0f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:19.582 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.594+0000 7f8bc2567700 1 -- 172.21.15.139:0/145519442 shutdown_connections 2021-11-20T06:08:19.582 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.594+0000 7f8bc2567700 1 -- 172.21.15.139:0/145519442 wait complete. 2021-11-20T06:08:19.582 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.594+0000 7f8bc2567700 1 Processor -- start 2021-11-20T06:08:19.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.595+0000 7f8bc2567700 1 -- start start 2021-11-20T06:08:19.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.595+0000 7f8bc2567700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bbc0f5b70 0x7f8bbc0f5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:19.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.595+0000 7f8bc2567700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8bbc0f6510 0x7f8bbc10bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:19.583 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.595+0000 7f8bc2567700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bbc10c150 con 0x7f8bbc0f6510 2021-11-20T06:08:19.584 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.595+0000 7f8bc2567700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bbc10c290 con 0x7f8bbc0f5b70 2021-11-20T06:08:19.584 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.595+0000 7f8bbb7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8bbc0f6510 0x7f8bbc10bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:19.584 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.595+0000 7f8bbb7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8bbc0f6510 0x7f8bbc10bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59200/0 (socket says 172.21.15.139:59200) 2021-11-20T06:08:19.584 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.595+0000 7f8bbb7fe700 1 -- 172.21.15.139:0/2271186333 learned_addr learned my addr 172.21.15.139:0/2271186333 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:19.584 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.595+0000 7f8bbbfff700 1 --2- 172.21.15.139:0/2271186333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bbc0f5b70 0x7f8bbc0f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:19.585 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.595+0000 7f8bbb7fe700 1 -- 172.21.15.139:0/2271186333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bbc0f5b70 msgr2=0x7f8bbc0f5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:19.585 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.595+0000 7f8bbb7fe700 1 --2- 172.21.15.139:0/2271186333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bbc0f5b70 0x7f8bbc0f5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:19.585 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.595+0000 7f8bbb7fe700 1 -- 172.21.15.139:0/2271186333 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac005040 con 0x7f8bbc0f6510 2021-11-20T06:08:19.585 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.595+0000 7f8bbb7fe700 1 --2- 172.21.15.139:0/2271186333 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8bbc0f6510 0x7f8bbc10bc60 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f8bb0002730 tx=0x7f8bb0009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:19.585 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.595+0000 7f8bb8ff9700 1 -- 172.21.15.139:0/2271186333 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bb0009320 con 0x7f8bbc0f6510 2021-11-20T06:08:19.586 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.596+0000 7f8bb8ff9700 1 -- 172.21.15.139:0/2271186333 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8bb000f040 con 0x7f8bbc0f6510 2021-11-20T06:08:19.586 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.596+0000 7f8bc2567700 1 -- 172.21.15.139:0/2271186333 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bbc10c4f0 con 0x7f8bbc0f6510 2021-11-20T06:08:19.586 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.596+0000 7f8bb8ff9700 1 -- 172.21.15.139:0/2271186333 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bb0014820 con 0x7f8bbc0f6510 2021-11-20T06:08:19.586 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.596+0000 7f8bc2567700 1 -- 172.21.15.139:0/2271186333 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bbc10cab0 con 0x7f8bbc0f6510 2021-11-20T06:08:19.587 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.596+0000 7f8bc2567700 1 -- 172.21.15.139:0/2271186333 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bbc027a00 con 0x7f8bbc0f6510 2021-11-20T06:08:19.588 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.597+0000 7f8bb8ff9700 1 -- 172.21.15.139:0/2271186333 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8bb00077b0 con 0x7f8bbc0f6510 2021-11-20T06:08:19.588 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.597+0000 7f8bb8ff9700 1 --2- 172.21.15.139:0/2271186333 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8ba405c2a0 0x7f8ba405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:19.588 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.597+0000 7f8bb8ff9700 1 -- 172.21.15.139:0/2271186333 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f8bb0076700 con 0x7f8bbc0f6510 2021-11-20T06:08:19.589 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.597+0000 7f8bbbfff700 1 --2- 172.21.15.139:0/2271186333 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8ba405c2a0 0x7f8ba405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:19.589 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.598+0000 7f8bbbfff700 1 --2- 172.21.15.139:0/2271186333 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8ba405c2a0 0x7f8ba405e760 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f8bac009f50 tx=0x7f8bac009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:19.589 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.601+0000 7f8bb8ff9700 1 -- 172.21.15.139:0/2271186333 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8bb0048e10 con 0x7f8bbc0f6510 2021-11-20T06:08:19.769 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.785+0000 7f8bc2567700 1 -- 172.21.15.139:0/2271186333 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bbc0fdd30 con 0x7f8bbc0f6510 2021-11-20T06:08:19.770 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.786+0000 7f8bb8ff9700 1 -- 172.21.15.139:0/2271186333 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2185 (secure 0 0 0) 0x7f8bb004c430 con 0x7f8bbc0f6510 2021-11-20T06:08:19.771 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:19.771 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24615,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/339439062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":339439062},{"type":"v1","addr":"172.21.15.178:6827","nonce":339439062}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":88}],"filesystems":[{"mdsmap":{"epoch":90,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:19.258180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24583},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":88,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:19.772 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.787+0000 7f8bc2567700 1 -- 172.21.15.139:0/2271186333 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8ba405c2a0 msgr2=0x7f8ba405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:19.773 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.787+0000 7f8bc2567700 1 --2- 172.21.15.139:0/2271186333 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8ba405c2a0 0x7f8ba405e760 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f8bac009f50 tx=0x7f8bac009b30).stop 2021-11-20T06:08:19.774 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.787+0000 7f8bc2567700 1 -- 172.21.15.139:0/2271186333 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8bbc0f6510 msgr2=0x7f8bbc10bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:19.774 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.787+0000 7f8bc2567700 1 --2- 172.21.15.139:0/2271186333 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8bbc0f6510 0x7f8bbc10bc60 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f8bb0002730 tx=0x7f8bb0009e00).stop 2021-11-20T06:08:19.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.787+0000 7f8bc2567700 1 -- 172.21.15.139:0/2271186333 shutdown_connections 2021-11-20T06:08:19.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.787+0000 7f8bc2567700 1 --2- 172.21.15.139:0/2271186333 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f8ba405c2a0 0x7f8ba405e760 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:19.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.787+0000 7f8bc2567700 1 --2- 172.21.15.139:0/2271186333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8bbc0f5b70 0x7f8bbc0f5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:19.775 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.787+0000 7f8bc2567700 1 --2- 172.21.15.139:0/2271186333 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8bbc0f6510 0x7f8bbc10bc60 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:19.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.787+0000 7f8bc2567700 1 -- 172.21.15.139:0/2271186333 >> 172.21.15.139:0/2271186333 conn(0x7f8bbc0f0440 msgr2=0x7f8bbc0fa410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:19.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.788+0000 7f8bc2567700 1 -- 172.21.15.139:0/2271186333 shutdown_connections 2021-11-20T06:08:19.776 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:19.788+0000 7f8bc2567700 1 -- 172.21.15.139:0/2271186333 wait complete. 2021-11-20T06:08:19.776 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 90 2021-11-20T06:08:19.784 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:19.258180+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24583}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 88, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/4081910640', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 4081910640}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 4081910640}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:19.784 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T06:08:20.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:20 smithi139 conmon[33501]: cluster 2021-11-20T06:08:19.263939+0000 mon.smithi139 ( 2021-11-20T06:08:20.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:20 smithi139 conmon[33501]: mon.0) 897 : cluster [DBG] mds.? [v2:172.21.15.178:6824/4081910640,v1:172.21.15.178:6825/4081910640] up:rejoin 2021-11-20T06:08:20.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:20 smithi139 conmon[33501]: cluster 2021-11-20T06:08:19.263987+0000 mon.smithi139 (mon.0) 898 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:rejoin} 1 up:standby 2021-11-20T06:08:20.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:20 smithi139 conmon[33501]: audit 2021-11-20T06:08:19.786456+0000 2021-11-20T06:08:20.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:20 smithi139 conmon[33501]: mon.smithi139 (mon.0) 899 : audit [DBG] from='client.? 172.21.15.139:0/2271186333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:20.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:20 smithi178 conmon[36600]: cluster 2021-11-20T06:08:19.263939+0000 mon.smithi139 (mon.0) 897 : cluster [DBG] 2021-11-20T06:08:20.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:20 smithi178 conmon[36600]: mds.? [v2:172.21.15.178:6824/4081910640,v1:172.21.15.178:6825/4081910640] up:rejoin 2021-11-20T06:08:20.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:20 smithi178 conmon[36600]: cluster 2021-11-20T06:08:19.263987+0000 mon.smithi139 (mon.0) 898 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:rejoin} 1 up:standby 2021-11-20T06:08:20.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:20 smithi178 conmon[36600]: audit 2021-11-20T06: 2021-11-20T06:08:20.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:20 smithi178 conmon[36600]: 08:19.786456+0000 mon.smithi139 (mon.0) 899 : audit [DBG] 2021-11-20T06:08:20.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:20 smithi178 conmon[36600]: from='client.? 172.21.15.139:0/2271186333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:20.785 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:20.906 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.919+0000 7f7f1cc1a700 1 -- 172.21.15.139:0/991136442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f180f6770 msgr2=0x7f7f180f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:20.907 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.919+0000 7f7f1cc1a700 1 --2- 172.21.15.139:0/991136442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f180f6770 0x7f7f180f6bd0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f7f0c004660 tx=0x7f7f0c009b30).stop 2021-11-20T06:08:20.907 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.919+0000 7f7f1cc1a700 1 -- 172.21.15.139:0/991136442 shutdown_connections 2021-11-20T06:08:20.907 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.919+0000 7f7f1cc1a700 1 --2- 172.21.15.139:0/991136442 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f180f6770 0x7f7f180f6bd0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:20.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.919+0000 7f7f1cc1a700 1 --2- 172.21.15.139:0/991136442 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f180f54d0 0x7f7f180f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-20T06:08:20.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.919+0000 7f7f1cc1a700 1 -- 172.21.15.139:0/991136442 >> 172.21.15.139:0/991136442 conn(0x7f7f180f0c10 msgr2=0x7f7f180f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:20.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.919+0000 7f7f1cc1a700 1 -- 172.21.15.139:0/991136442 shutdown_connections 2021-11-20T06:08:20.908 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.919+0000 7f7f1cc1a700 1 -- 172.21.15.139:0/991136442 wait complete. 2021-11-20T06:08:20.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.920+0000 7f7f1cc1a700 1 Processor -- start 2021-11-20T06:08:20.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.920+0000 7f7f1cc1a700 1 -- start start 2021-11-20T06:08:20.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.920+0000 7f7f1cc1a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f180f54d0 0x7f7f18109a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:20.909 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.920+0000 7f7f1cc1a700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f180f6770 0x7f7f1810bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:20.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.920+0000 7f7f1cc1a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f180605f0 con 0x7f7f180f54d0 2021-11-20T06:08:20.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.920+0000 7f7f1cc1a700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f18060730 con 0x7f7f180f6770 2021-11-20T06:08:20.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.920+0000 7f7f177fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f180f54d0 0x7f7f18109a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:20.910 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.920+0000 7f7f177fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f180f54d0 0x7f7f18109a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59212/0 (socket says 172.21.15.139:59212) 2021-11-20T06:08:20.911 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.920+0000 7f7f177fe700 1 -- 172.21.15.139:0/660161414 learned_addr learned my addr 172.21.15.139:0/660161414 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:20.911 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.920+0000 7f7f177fe700 1 -- 172.21.15.139:0/660161414 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f180f6770 msgr2=0x7f7f1810bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:20.911 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.920+0000 7f7f177fe700 1 --2- 172.21.15.139:0/660161414 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f180f6770 0x7f7f1810bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:08:20.911 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.920+0000 7f7f177fe700 1 -- 172.21.15.139:0/660161414 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c005040 con 0x7f7f180f54d0 2021-11-20T06:08:20.912 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.920+0000 7f7f177fe700 1 --2- 172.21.15.139:0/660161414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f180f54d0 0x7f7f18109a30 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f7f080093f0 tx=0x7f7f08006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:20.912 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.921+0000 7f7efffff700 1 -- 172.21.15.139:0/660161414 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f08009ea0 con 0x7f7f180f54d0 2021-11-20T06:08:20.913 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.921+0000 7f7f1cc1a700 1 -- 172.21.15.139:0/660161414 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1810c590 con 0x7f7f180f54d0 2021-11-20T06:08:20.913 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.921+0000 7f7efffff700 1 -- 172.21.15.139:0/660161414 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f08013af0 con 0x7f7f180f54d0 2021-11-20T06:08:20.914 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.921+0000 7f7efffff700 1 -- 172.21.15.139:0/660161414 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f0800dad0 con 0x7f7f180f54d0 2021-11-20T06:08:20.914 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.921+0000 7f7f1cc1a700 1 -- 172.21.15.139:0/660161414 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f1810cb20 con 0x7f7f180f54d0 2021-11-20T06:08:20.914 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.921+0000 7f7f1cc1a700 1 -- 172.21.15.139:0/660161414 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f18045ad0 con 0x7f7f180f54d0 2021-11-20T06:08:20.914 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.922+0000 7f7efffff700 1 -- 172.21.15.139:0/660161414 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7f0801a060 con 0x7f7f180f54d0 2021-11-20T06:08:20.915 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.922+0000 7f7efffff700 1 --2- 172.21.15.139:0/660161414 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7f0005c350 0x7f7f0005e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:20.915 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.922+0000 7f7efffff700 1 -- 172.21.15.139:0/660161414 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f7f08076570 con 0x7f7f180f54d0 2021-11-20T06:08:20.915 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.922+0000 7f7f16ffd700 1 --2- 172.21.15.139:0/660161414 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7f0005c350 0x7f7f0005e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:20.917 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.923+0000 7f7f16ffd700 1 --2- 172.21.15.139:0/660161414 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7f0005c350 0x7f7f0005e810 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f7f0c009f50 tx=0x7f7f0c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:20.917 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:20.926+0000 7f7efffff700 1 -- 172.21.15.139:0/660161414 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f08048b60 con 0x7f7f180f54d0 2021-11-20T06:08:21.106 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:21.121+0000 7f7f1cc1a700 1 -- 172.21.15.139:0/660161414 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f18027a00 con 0x7f7f180f54d0 2021-11-20T06:08:21.107 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:21.122+0000 7f7efffff700 1 -- 172.21.15.139:0/660161414 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2185 (secure 0 0 0) 0x7f7f0804c180 con 0x7f7f180f54d0 2021-11-20T06:08:21.107 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:21.107 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24615,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.178:6827/339439062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":339439062},{"type":"v1","addr":"172.21.15.178:6827","nonce":339439062}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":88}],"filesystems":[{"mdsmap":{"epoch":90,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:19.258180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24583},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":88,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:21.109 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:21.124+0000 7f7f1cc1a700 1 -- 172.21.15.139:0/660161414 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7f0005c350 msgr2=0x7f7f0005e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:21.109 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:21.124+0000 7f7f1cc1a700 1 --2- 172.21.15.139:0/660161414 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7f0005c350 0x7f7f0005e810 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f7f0c009f50 tx=0x7f7f0c009b30).stop 2021-11-20T06:08:21.109 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:21.124+0000 7f7f1cc1a700 1 -- 172.21.15.139:0/660161414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f180f54d0 msgr2=0x7f7f18109a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:21.109 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:21.124+0000 7f7f1cc1a700 1 --2- 172.21.15.139:0/660161414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f180f54d0 0x7f7f18109a30 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f7f080093f0 tx=0x7f7f08006ec0).stop 2021-11-20T06:08:21.110 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:21.124+0000 7f7f1cc1a700 1 -- 172.21.15.139:0/660161414 shutdown_connections 2021-11-20T06:08:21.110 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:21.124+0000 7f7f1cc1a700 1 --2- 172.21.15.139:0/660161414 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f7f0005c350 0x7f7f0005e810 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:21.110 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:21.124+0000 7f7f1cc1a700 1 --2- 172.21.15.139:0/660161414 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7f180f54d0 0x7f7f18109a30 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:21.110 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:21.124+0000 7f7f1cc1a700 1 --2- 172.21.15.139:0/660161414 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f180f6770 0x7f7f1810bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:08:21.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:21.124+0000 7f7f1cc1a700 1 -- 172.21.15.139:0/660161414 >> 172.21.15.139:0/660161414 conn(0x7f7f180f0c10 msgr2=0x7f7f180fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:21.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:21.124+0000 7f7f1cc1a700 1 -- 172.21.15.139:0/660161414 shutdown_connections 2021-11-20T06:08:21.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:21.124+0000 7f7f1cc1a700 1 -- 172.21.15.139:0/660161414 wait complete. 2021-11-20T06:08:21.111 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 90 2021-11-20T06:08:21.120 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:19.258180+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24583}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 88, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/4081910640', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 4081910640}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 4081910640}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:21.121 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T06:08:21.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:21 smithi139 conmon[33501]: cluster 2021-11-20T06:08:20.282908+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:08:21.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:21 smithi139 conmon[33501]: ) 336 : cluster [DBG] pgmap v314: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T06:08:21.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:21 smithi139 conmon[33501]: audit 2021-11-20T06:08:21.122863+0000 mon.smithi139 (mon.0) 900 : audit [DBG] from='client.? 172.21.15.139:0/660161414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:21.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:21 smithi178 conmon[36600]: cluster 2021-11-20T06:08:20.282908+0000 mgr.smithi139.dplyxn (mgr.14162) 336 : cluster 2021-11-20T06:08:21.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:21 smithi178 conmon[36600]: [DBG] pgmap v314: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 819 B/s rd, 0 op/s 2021-11-20T06:08:21.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:21 smithi178 conmon[36600]: audit 2021-11-20T06:08:21.122863+0000 mon.smithi139 (mon.0) 900 : audit [DBG] from='client.? 172.21.15.139:0/660161414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:22.122 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:22.255 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.268+0000 7fc5db59e700 1 -- 172.21.15.139:0/3281669528 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5dc0f54d0 msgr2=0x7fc5dc0f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:22.255 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.268+0000 7fc5db59e700 1 --2- 172.21.15.139:0/3281669528 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5dc0f54d0 0x7fc5dc0f58f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7fc5cc004660 tx=0x7fc5cc009b30).stop 2021-11-20T06:08:22.255 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.268+0000 7fc5db59e700 1 -- 172.21.15.139:0/3281669528 shutdown_connections 2021-11-20T06:08:22.256 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.268+0000 7fc5db59e700 1 --2- 172.21.15.139:0/3281669528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5dc0f6770 0x7fc5dc0f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:22.256 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.268+0000 7fc5db59e700 1 --2- 172.21.15.139:0/3281669528 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5dc0f54d0 0x7fc5dc0f58f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:22.256 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.268+0000 7fc5db59e700 1 -- 172.21.15.139:0/3281669528 >> 172.21.15.139:0/3281669528 conn(0x7fc5dc0f0c10 msgr2=0x7fc5dc0f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:22.256 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.268+0000 7fc5db59e700 1 -- 172.21.15.139:0/3281669528 shutdown_connections 2021-11-20T06:08:22.256 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.268+0000 7fc5db59e700 1 -- 172.21.15.139:0/3281669528 wait complete. 2021-11-20T06:08:22.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.268+0000 7fc5db59e700 1 Processor -- start 2021-11-20T06:08:22.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.268+0000 7fc5db59e700 1 -- start start 2021-11-20T06:08:22.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.268+0000 7fc5db59e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5dc0f54d0 0x7fc5dc105cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:22.257 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.268+0000 7fc5db59e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5dc0f6770 0x7fc5dc1061e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:22.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.268+0000 7fc5db59e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc106790 con 0x7fc5dc0f54d0 2021-11-20T06:08:22.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.268+0000 7fc5db59e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc106f90 con 0x7fc5dc0f6770 2021-11-20T06:08:22.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.269+0000 7fc5da59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5dc0f54d0 0x7fc5dc105cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:22.258 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.269+0000 7fc5da59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5dc0f54d0 0x7fc5dc105cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59226/0 (socket says 172.21.15.139:59226) 2021-11-20T06:08:22.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.269+0000 7fc5da59c700 1 -- 172.21.15.139:0/844915850 learned_addr learned my addr 172.21.15.139:0/844915850 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:22.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.269+0000 7fc5d9d9b700 1 --2- 172.21.15.139:0/844915850 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5dc0f6770 0x7fc5dc1061e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:22.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.269+0000 7fc5da59c700 1 -- 172.21.15.139:0/844915850 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5dc0f6770 msgr2=0x7fc5dc1061e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:22.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.269+0000 7fc5da59c700 1 --2- 172.21.15.139:0/844915850 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5dc0f6770 0x7fc5dc1061e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:22.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.269+0000 7fc5da59c700 1 -- 172.21.15.139:0/844915850 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc005040 con 0x7fc5dc0f54d0 2021-11-20T06:08:22.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.269+0000 7fc5d9d9b700 1 --2- 172.21.15.139:0/844915850 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5dc0f6770 0x7fc5dc1061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:08:22.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.269+0000 7fc5da59c700 1 --2- 172.21.15.139:0/844915850 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5dc0f54d0 0x7fc5dc105cf0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fc5cc004e30 tx=0x7fc5cc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:22.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.269+0000 7fc5caffd700 1 -- 172.21.15.139:0/844915850 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5cc018070 con 0x7fc5dc0f54d0 2021-11-20T06:08:22.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.269+0000 7fc5db59e700 1 -- 172.21.15.139:0/844915850 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5dc107210 con 0x7fc5dc0f54d0 2021-11-20T06:08:22.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.269+0000 7fc5db59e700 1 -- 172.21.15.139:0/844915850 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5dc107770 con 0x7fc5dc0f54d0 2021-11-20T06:08:22.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.270+0000 7fc5caffd700 1 -- 172.21.15.139:0/844915850 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5cc013410 con 0x7fc5dc0f54d0 2021-11-20T06:08:22.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.270+0000 7fc5caffd700 1 -- 172.21.15.139:0/844915850 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5cc01dc40 con 0x7fc5dc0f54d0 2021-11-20T06:08:22.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.270+0000 7fc5db59e700 1 -- 172.21.15.139:0/844915850 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c0004fa0 con 0x7fc5dc0f54d0 2021-11-20T06:08:22.263 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.271+0000 7fc5caffd700 1 -- 172.21.15.139:0/844915850 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc5cc01c3c0 con 0x7fc5dc0f54d0 2021-11-20T06:08:22.264 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.271+0000 7fc5caffd700 1 --2- 172.21.15.139:0/844915850 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc5c405c300 0x7fc5c405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:22.264 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.271+0000 7fc5caffd700 1 -- 172.21.15.139:0/844915850 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7fc5cc078040 con 0x7fc5dc0f54d0 2021-11-20T06:08:22.264 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.272+0000 7fc5d9d9b700 1 --2- 172.21.15.139:0/844915850 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc5c405c300 0x7fc5c405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:22.264 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.272+0000 7fc5d9d9b700 1 --2- 172.21.15.139:0/844915850 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc5c405c300 0x7fc5c405e7c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc5d0009740 tx=0x7fc5d0009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:22.265 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.278+0000 7fc5caffd700 1 -- 172.21.15.139:0/844915850 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5cc04a6a0 con 0x7fc5dc0f54d0 2021-11-20T06:08:22.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:22 smithi178 conmon[36600]: cluster 2021-11-20T06:08:21.271842+0000 mon.smithi139 (mon.0) 901 : cluster 2021-11-20T06:08:22.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:22 smithi178 conmon[36600]: [DBG] mds.? [v2:172.21.15.178:6826/339439062,v1:172.21.15.178:6827/339439062] up:standby 2021-11-20T06:08:22.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:22 smithi178 conmon[36600]: cluster 2021-11-20T06:08:21.271893+0000 mon.smithi139 (mon.0) 902 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:rejoin} 1 up:standby 2021-11-20T06:08:22.447 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.463+0000 7fc5db59e700 1 -- 172.21.15.139:0/844915850 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5c0005bf0 con 0x7fc5dc0f54d0 2021-11-20T06:08:22.448 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.464+0000 7fc5caffd700 1 -- 172.21.15.139:0/844915850 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+2184 (secure 0 0 0) 0x7fc5cc04dcc0 con 0x7fc5dc0f54d0 2021-11-20T06:08:22.448 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:22.449 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24615,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6827/339439062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":339439062},{"type":"v1","addr":"172.21.15.178:6827","nonce":339439062}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:19.258180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24583},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":88,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:22.451 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.465+0000 7fc5db59e700 1 -- 172.21.15.139:0/844915850 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc5c405c300 msgr2=0x7fc5c405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:22.451 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.465+0000 7fc5db59e700 1 --2- 172.21.15.139:0/844915850 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc5c405c300 0x7fc5c405e7c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc5d0009740 tx=0x7fc5d0009be0).stop 2021-11-20T06:08:22.451 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.465+0000 7fc5db59e700 1 -- 172.21.15.139:0/844915850 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5dc0f54d0 msgr2=0x7fc5dc105cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:22.452 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.465+0000 7fc5db59e700 1 --2- 172.21.15.139:0/844915850 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5dc0f54d0 0x7fc5dc105cf0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fc5cc004e30 tx=0x7fc5cc009b30).stop 2021-11-20T06:08:22.452 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.465+0000 7fc5db59e700 1 -- 172.21.15.139:0/844915850 shutdown_connections 2021-11-20T06:08:22.452 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.465+0000 7fc5db59e700 1 --2- 172.21.15.139:0/844915850 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc5c405c300 0x7fc5c405e7c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:22.452 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.465+0000 7fc5db59e700 1 --2- 172.21.15.139:0/844915850 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc5dc0f54d0 0x7fc5dc105cf0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:22.452 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.465+0000 7fc5db59e700 1 --2- 172.21.15.139:0/844915850 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5dc0f6770 0x7fc5dc1061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:22.453 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.465+0000 7fc5db59e700 1 -- 172.21.15.139:0/844915850 >> 172.21.15.139:0/844915850 conn(0x7fc5dc0f0c10 msgr2=0x7fc5dc0fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:22.453 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.465+0000 7fc5db59e700 1 -- 172.21.15.139:0/844915850 shutdown_connections 2021-11-20T06:08:22.453 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:22.465+0000 7fc5db59e700 1 -- 172.21.15.139:0/844915850 wait complete. 2021-11-20T06:08:22.453 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 91 2021-11-20T06:08:22.461 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:22 smithi139 conmon[33501]: cluster 2021-11-20T06:08:21 2021-11-20T06:08:22.461 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:22 smithi139 conmon[33501]: .271842+0000 mon.smithi139 (mon.0) 901 : cluster [DBG] mds.? [v2:172.21.15.178:6826/339439062,v1:172.21.15.178:6827/339439062] up:standby 2021-11-20T06:08:22.461 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:22 smithi139 conmon[33501]: cluster 2021-11-20T06:08:22.462 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:22 smithi139 conmon[33501]: 2021-11-20T06:08:21.271893+0000 mon.smithi139 ( 2021-11-20T06:08:22.463 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:22 smithi139 conmon[33501]: mon.0) 902 : cluster 2021-11-20T06:08:22.463 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:22 smithi139 conmon[33501]: [DBG] fsmap cephfs:1/1 {0=cephfs.smithi178.blvkbx=up:rejoin} 1 up:standby 2021-11-20T06:08:22.464 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:19.258180+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24583}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 88, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/4081910640', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 4081910640}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 4081910640}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:22.464 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T06:08:23.465 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:23.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:23 smithi139 conmon[33501]: cluster 2021-11-20T06:08:22.283346+0000 mgr.smithi139.dplyxn (mgr.14162) 337 : cluster 2021-11-20T06:08:23.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:23 smithi139 conmon[33501]: [DBG] pgmap v315: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:08:23.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:23 smithi139 conmon[33501]: audit 2021-11-20T06:08:22.464555+0000 mon.smithi139 (mon.0) 903 : audit [DBG] from='client.? 172.21.15.139:0/844915850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:23.594 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.607+0000 7fa8e99f9700 1 -- 172.21.15.139:0/849132352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa8e40f65d0 msgr2=0x7fa8e40f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:23.594 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.607+0000 7fa8e99f9700 1 --2- 172.21.15.139:0/849132352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa8e40f65d0 0x7fa8e40f69f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7fa8d8004660 tx=0x7fa8d8009b30).stop 2021-11-20T06:08:23.595 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.608+0000 7fa8e99f9700 1 -- 172.21.15.139:0/849132352 shutdown_connections 2021-11-20T06:08:23.595 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.608+0000 7fa8e99f9700 1 --2- 172.21.15.139:0/849132352 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8e40f7870 0x7fa8e40f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:23.595 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.608+0000 7fa8e99f9700 1 --2- 172.21.15.139:0/849132352 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa8e40f65d0 0x7fa8e40f69f0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:23.595 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.608+0000 7fa8e99f9700 1 -- 172.21.15.139:0/849132352 >> 172.21.15.139:0/849132352 conn(0x7fa8e40f1d10 msgr2=0x7fa8e40f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:23.596 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.608+0000 7fa8e99f9700 1 -- 172.21.15.139:0/849132352 shutdown_connections 2021-11-20T06:08:23.596 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.608+0000 7fa8e99f9700 1 -- 172.21.15.139:0/849132352 wait complete. 2021-11-20T06:08:23.596 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.608+0000 7fa8e99f9700 1 Processor -- start 2021-11-20T06:08:23.596 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.608+0000 7fa8e99f9700 1 -- start start 2021-11-20T06:08:23.597 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.608+0000 7fa8e99f9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa8e40f65d0 0x7fa8e4106da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:23.597 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.608+0000 7fa8e99f9700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8e40f7870 0x7fa8e4107290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:23.597 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.608+0000 7fa8e99f9700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e4107890 con 0x7fa8e40f65d0 2021-11-20T06:08:23.597 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.608+0000 7fa8e99f9700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e41080d0 con 0x7fa8e40f7870 2021-11-20T06:08:23.597 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.609+0000 7fa8e89f7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa8e40f65d0 0x7fa8e4106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:23.598 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.609+0000 7fa8e89f7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa8e40f65d0 0x7fa8e4106da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59236/0 (socket says 172.21.15.139:59236) 2021-11-20T06:08:23.598 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.609+0000 7fa8e89f7700 1 -- 172.21.15.139:0/1901713679 learned_addr learned my addr 172.21.15.139:0/1901713679 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:23.598 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.609+0000 7fa8e89f7700 1 -- 172.21.15.139:0/1901713679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8e40f7870 msgr2=0x7fa8e4107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:23.598 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.609+0000 7fa8e89f7700 1 --2- 172.21.15.139:0/1901713679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8e40f7870 0x7fa8e4107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:08:23.599 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.609+0000 7fa8e89f7700 1 -- 172.21.15.139:0/1901713679 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d8005040 con 0x7fa8e40f65d0 2021-11-20T06:08:23.599 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.609+0000 7fa8e89f7700 1 --2- 172.21.15.139:0/1901713679 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa8e40f65d0 0x7fa8e4106da0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7fa8d8009f50 tx=0x7fa8d8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:23.599 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.609+0000 7fa8e17fa700 1 -- 172.21.15.139:0/1901713679 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8d8018070 con 0x7fa8e40f65d0 2021-11-20T06:08:23.599 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.609+0000 7fa8e17fa700 1 -- 172.21.15.139:0/1901713679 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8d8013410 con 0x7fa8e40f65d0 2021-11-20T06:08:23.600 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.609+0000 7fa8e99f9700 1 -- 172.21.15.139:0/1901713679 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e4108350 con 0x7fa8e40f65d0 2021-11-20T06:08:23.600 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.609+0000 7fa8e17fa700 1 -- 172.21.15.139:0/1901713679 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8d801c620 con 0x7fa8e40f65d0 2021-11-20T06:08:23.600 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.609+0000 7fa8e99f9700 1 -- 172.21.15.139:0/1901713679 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e4108830 con 0x7fa8e40f65d0 2021-11-20T06:08:23.600 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.610+0000 7fa8e99f9700 1 -- 172.21.15.139:0/1901713679 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e40f59c0 con 0x7fa8e40f65d0 2021-11-20T06:08:23.602 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.610+0000 7fa8e17fa700 1 -- 172.21.15.139:0/1901713679 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa8d801d420 con 0x7fa8e40f65d0 2021-11-20T06:08:23.602 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.611+0000 7fa8e17fa700 1 --2- 172.21.15.139:0/1901713679 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa8d405c300 0x7fa8d405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:23.602 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.611+0000 7fa8e17fa700 1 -- 172.21.15.139:0/1901713679 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7fa8d8078240 con 0x7fa8e40f65d0 2021-11-20T06:08:23.602 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.611+0000 7fa8e3fff700 1 --2- 172.21.15.139:0/1901713679 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa8d405c300 0x7fa8d405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:23.603 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.611+0000 7fa8e3fff700 1 --2- 172.21.15.139:0/1901713679 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa8d405c300 0x7fa8d405e7c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fa8e4106b30 tx=0x7fa8d0007a40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:23.603 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.615+0000 7fa8e17fa700 1 -- 172.21.15.139:0/1901713679 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8d804a9d0 con 0x7fa8e40f65d0 2021-11-20T06:08:23.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:23 smithi178 conmon[36600]: cluster 2021-11-20T06:08:22.283346+0000 mgr.smithi139.dplyxn (mgr.14162) 337 : cluster 2021-11-20T06:08:23.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:23 smithi178 conmon[36600]: [DBG] pgmap v315: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:08:23.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:23 smithi178 conmon[36600]: audit 2021-11-20T06:08:22.464555+0000 mon.smithi139 (mon.0) 903 : audit [DBG] from='client.? 172.21.15.139:0/844915850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:23.783 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.798+0000 7fa8e99f9700 1 -- 172.21.15.139:0/1901713679 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8e4045ad0 con 0x7fa8e40f65d0 2021-11-20T06:08:23.783 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.798+0000 7fa8e17fa700 1 -- 172.21.15.139:0/1901713679 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+2184 (secure 0 0 0) 0x7fa8d804dff0 con 0x7fa8e40f65d0 2021-11-20T06:08:23.784 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:23.784 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24615,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6827/339439062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":339439062},{"type":"v1","addr":"172.21.15.178:6827","nonce":339439062}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:19.258180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24583},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":88,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:23.785 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.800+0000 7fa8e99f9700 1 -- 172.21.15.139:0/1901713679 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa8d405c300 msgr2=0x7fa8d405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:23.785 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.800+0000 7fa8e99f9700 1 --2- 172.21.15.139:0/1901713679 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa8d405c300 0x7fa8d405e7c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fa8e4106b30 tx=0x7fa8d0007a40).stop 2021-11-20T06:08:23.786 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.800+0000 7fa8e99f9700 1 -- 172.21.15.139:0/1901713679 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa8e40f65d0 msgr2=0x7fa8e4106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:23.787 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.800+0000 7fa8e99f9700 1 --2- 172.21.15.139:0/1901713679 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa8e40f65d0 0x7fa8e4106da0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7fa8d8009f50 tx=0x7fa8d8009b30).stop 2021-11-20T06:08:23.787 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.800+0000 7fa8e99f9700 1 -- 172.21.15.139:0/1901713679 shutdown_connections 2021-11-20T06:08:23.788 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.800+0000 7fa8e99f9700 1 --2- 172.21.15.139:0/1901713679 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fa8d405c300 0x7fa8d405e7c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:23.788 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.800+0000 7fa8e99f9700 1 --2- 172.21.15.139:0/1901713679 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa8e40f65d0 0x7fa8e4106da0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:23.788 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.800+0000 7fa8e99f9700 1 --2- 172.21.15.139:0/1901713679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8e40f7870 0x7fa8e4107290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-20T06:08:23.788 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.800+0000 7fa8e99f9700 1 -- 172.21.15.139:0/1901713679 >> 172.21.15.139:0/1901713679 conn(0x7fa8e40f1d10 msgr2=0x7fa8e40fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:23.788 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.800+0000 7fa8e99f9700 1 -- 172.21.15.139:0/1901713679 shutdown_connections 2021-11-20T06:08:23.789 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:23.800+0000 7fa8e99f9700 1 -- 172.21.15.139:0/1901713679 wait complete. 2021-11-20T06:08:23.789 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 91 2021-11-20T06:08:23.797 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:19.258180+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24583}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 88, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/4081910640', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 4081910640}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 4081910640}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:23.797 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T06:08:24.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:24 smithi139 conmon[33501]: audit 2021-11-20T06:08:23.799400+0000 mon.smithi139 (mon.0) 2021-11-20T06:08:24.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:24 smithi139 conmon[33501]: 904 : audit [DBG] from='client.? 172.21.15.139:0/1901713679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:24.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:24 smithi178 conmon[36600]: audit 2021-11-20T06:08:23.799400+0000 mon.smithi139 (mon 2021-11-20T06:08:24.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:24 smithi178 conmon[36600]: .0) 904 : audit [DBG] from='client.? 172.21.15.139:0/1901713679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:24.798 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:24.930 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.940+0000 7f90ef59e700 1 -- 172.21.15.139:0/3258459704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90f00f4c00 msgr2=0x7f90f00f5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:24.930 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.940+0000 7f90ef59e700 1 --2- 172.21.15.139:0/3258459704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90f00f4c00 0x7f90f00f5020 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f90e0004660 tx=0x7f90e0009b30).stop 2021-11-20T06:08:24.931 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90ef59e700 1 -- 172.21.15.139:0/3258459704 shutdown_connections 2021-11-20T06:08:24.932 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90ef59e700 1 --2- 172.21.15.139:0/3258459704 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90f00f5ea0 0x7f90f00f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:24.932 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90ef59e700 1 --2- 172.21.15.139:0/3258459704 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90f00f4c00 0x7f90f00f5020 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:24.933 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90ef59e700 1 -- 172.21.15.139:0/3258459704 >> 172.21.15.139:0/3258459704 conn(0x7f90f00f0360 msgr2=0x7f90f00f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:24.933 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90ef59e700 1 -- 172.21.15.139:0/3258459704 shutdown_connections 2021-11-20T06:08:24.933 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90ef59e700 1 -- 172.21.15.139:0/3258459704 wait complete. 2021-11-20T06:08:24.933 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90ef59e700 1 Processor -- start 2021-11-20T06:08:24.933 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90ef59e700 1 -- start start 2021-11-20T06:08:24.934 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90ef59e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90f00f4c00 0x7f90f0106dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:24.934 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90ef59e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90f00f5ea0 0x7f90f01072b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:24.934 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90ef59e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f01078b0 con 0x7f90f00f4c00 2021-11-20T06:08:24.934 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90ef59e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f01080f0 con 0x7f90f00f5ea0 2021-11-20T06:08:24.935 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90ee59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90f00f4c00 0x7f90f0106dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:24.935 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90ee59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90f00f4c00 0x7f90f0106dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59248/0 (socket says 172.21.15.139:59248) 2021-11-20T06:08:24.935 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90ee59c700 1 -- 172.21.15.139:0/405432153 learned_addr learned my addr 172.21.15.139:0/405432153 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:24.935 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.941+0000 7f90edd9b700 1 --2- 172.21.15.139:0/405432153 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90f00f5ea0 0x7f90f01072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:24.935 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.942+0000 7f90ee59c700 1 -- 172.21.15.139:0/405432153 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90f00f5ea0 msgr2=0x7f90f01072b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:24.936 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.942+0000 7f90ee59c700 1 --2- 172.21.15.139:0/405432153 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90f00f5ea0 0x7f90f01072b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:24.936 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.942+0000 7f90ee59c700 1 -- 172.21.15.139:0/405432153 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e0005040 con 0x7f90f00f4c00 2021-11-20T06:08:24.936 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.942+0000 7f90edd9b700 1 --2- 172.21.15.139:0/405432153 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90f00f5ea0 0x7f90f01072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-20T06:08:24.936 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.942+0000 7f90ee59c700 1 --2- 172.21.15.139:0/405432153 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90f00f4c00 0x7f90f0106dc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f90e0007a60 tx=0x7f90e0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:24.937 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.942+0000 7f90deffd700 1 -- 172.21.15.139:0/405432153 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90e0018070 con 0x7f90f00f4c00 2021-11-20T06:08:24.937 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.942+0000 7f90deffd700 1 -- 172.21.15.139:0/405432153 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90e0013410 con 0x7f90f00f4c00 2021-11-20T06:08:24.937 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.942+0000 7f90ef59e700 1 -- 172.21.15.139:0/405432153 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f0108370 con 0x7f90f00f4c00 2021-11-20T06:08:24.937 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.942+0000 7f90deffd700 1 -- 172.21.15.139:0/405432153 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90e001c620 con 0x7f90f00f4c00 2021-11-20T06:08:24.937 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.942+0000 7f90ef59e700 1 -- 172.21.15.139:0/405432153 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f0108850 con 0x7f90f00f4c00 2021-11-20T06:08:24.938 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.943+0000 7f90ef59e700 1 -- 172.21.15.139:0/405432153 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f00f3ff0 con 0x7f90f00f4c00 2021-11-20T06:08:24.938 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.943+0000 7f90deffd700 1 -- 172.21.15.139:0/405432153 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f90e001d420 con 0x7f90f00f4c00 2021-11-20T06:08:24.938 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.943+0000 7f90deffd700 1 --2- 172.21.15.139:0/405432153 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f90d805c2f0 0x7f90d805e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:24.938 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.944+0000 7f90deffd700 1 -- 172.21.15.139:0/405432153 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f90e0078230 con 0x7f90f00f4c00 2021-11-20T06:08:24.939 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.944+0000 7f90edd9b700 1 --2- 172.21.15.139:0/405432153 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f90d805c2f0 0x7f90d805e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:24.939 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.944+0000 7f90edd9b700 1 --2- 172.21.15.139:0/405432153 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f90d805c2f0 0x7f90d805e7b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f90f0106b50 tx=0x7f90e4009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:24.939 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:24.947+0000 7f90deffd700 1 -- 172.21.15.139:0/405432153 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f90e004a9c0 con 0x7f90f00f4c00 2021-11-20T06:08:25.114 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:25.130+0000 7f90ef59e700 1 -- 172.21.15.139:0/405432153 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90f0027b20 con 0x7f90f00f4c00 2021-11-20T06:08:25.115 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:25.131+0000 7f90deffd700 1 -- 172.21.15.139:0/405432153 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+2184 (secure 0 0 0) 0x7f90e004dfe0 con 0x7f90f00f4c00 2021-11-20T06:08:25.116 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:25.116 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24615,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6827/339439062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":339439062},{"type":"v1","addr":"172.21.15.178:6827","nonce":339439062}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:19.258180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24583},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":88,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:25.118 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:25.132+0000 7f90ef59e700 1 -- 172.21.15.139:0/405432153 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f90d805c2f0 msgr2=0x7f90d805e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:25.119 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:25.132+0000 7f90ef59e700 1 --2- 172.21.15.139:0/405432153 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f90d805c2f0 0x7f90d805e7b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f90f0106b50 tx=0x7f90e4009be0).stop 2021-11-20T06:08:25.119 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:25.132+0000 7f90ef59e700 1 -- 172.21.15.139:0/405432153 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90f00f4c00 msgr2=0x7f90f0106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:25.119 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:25.132+0000 7f90ef59e700 1 --2- 172.21.15.139:0/405432153 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90f00f4c00 0x7f90f0106dc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f90e0007a60 tx=0x7f90e0009b30).stop 2021-11-20T06:08:25.120 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:25.132+0000 7f90ef59e700 1 -- 172.21.15.139:0/405432153 shutdown_connections 2021-11-20T06:08:25.120 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:25.132+0000 7f90ef59e700 1 --2- 172.21.15.139:0/405432153 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f90d805c2f0 0x7f90d805e7b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:25.120 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:25.132+0000 7f90ef59e700 1 --2- 172.21.15.139:0/405432153 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90f00f4c00 0x7f90f0106dc0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:25.120 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:25.132+0000 7f90ef59e700 1 --2- 172.21.15.139:0/405432153 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90f00f5ea0 0x7f90f01072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:25.121 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:25.132+0000 7f90ef59e700 1 -- 172.21.15.139:0/405432153 >> 172.21.15.139:0/405432153 conn(0x7f90f00f0360 msgr2=0x7f90f00fa3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:25.121 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:25.132+0000 7f90ef59e700 1 -- 172.21.15.139:0/405432153 shutdown_connections 2021-11-20T06:08:25.121 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:25.132+0000 7f90ef59e700 1 -- 172.21.15.139:0/405432153 wait complete. 2021-11-20T06:08:25.121 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 91 2021-11-20T06:08:25.129 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:19.258180+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24583}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 88, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/4081910640', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 4081910640}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 4081910640}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:25.129 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T06:08:25.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:25 smithi139 conmon[33501]: cluster 2021-11-20T06:08 2021-11-20T06:08:25.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:25 smithi139 conmon[33501]: :24.283785+0000 mgr.smithi139.dplyxn (mgr.14162) 338 : cluster [DBG] pgmap v316: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:08:25.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:25 smithi139 conmon[33501]: audit 2021-11-20T06:08:25.131506+0000 mon.smithi139 (mon.0 2021-11-20T06:08:25.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:25 smithi139 conmon[33501]: ) 905 : audit [DBG] from='client.? 172.21.15.139:0/405432153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:25.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:25 smithi178 conmon[36600]: cluster 2021-11-20T06:08:24.283785+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:08:25.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:25 smithi178 conmon[36600]: ) 338 : cluster [DBG] pgmap v316: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:08:25.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:25 smithi178 conmon[36600]: audit 2021-11-20T06:08:25.131506+0000 mon.smithi139 (mon.0) 905 : audit [DBG] 2021-11-20T06:08:25.693 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:25 smithi178 conmon[36600]: from='client.? 172.21.15.139:0/405432153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:26.130 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:26.259 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.273+0000 7f673a7df700 1 -- 172.21.15.139:0/2022665583 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67340f53a0 msgr2=0x7f67340f7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:26.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.273+0000 7f673a7df700 1 --2- 172.21.15.139:0/2022665583 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67340f53a0 0x7f67340f7ce0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f6728004660 tx=0x7f6728009b30).stop 2021-11-20T06:08:26.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.273+0000 7f673a7df700 1 -- 172.21.15.139:0/2022665583 shutdown_connections 2021-11-20T06:08:26.260 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.273+0000 7f673a7df700 1 --2- 172.21.15.139:0/2022665583 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67340f53a0 0x7f67340f7ce0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:26.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.273+0000 7f673a7df700 1 --2- 172.21.15.139:0/2022665583 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67340f4a00 0x7f67340f4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:26.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.273+0000 7f673a7df700 1 -- 172.21.15.139:0/2022665583 >> 172.21.15.139:0/2022665583 conn(0x7f67340f0440 msgr2=0x7f67340f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:26.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.273+0000 7f673a7df700 1 -- 172.21.15.139:0/2022665583 shutdown_connections 2021-11-20T06:08:26.261 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.273+0000 7f673a7df700 1 -- 172.21.15.139:0/2022665583 wait complete. 2021-11-20T06:08:26.262 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.273+0000 7f673a7df700 1 Processor -- start 2021-11-20T06:08:26.262 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.273+0000 7f673a7df700 1 -- start start 2021-11-20T06:08:26.263 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.273+0000 7f673a7df700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67340f4a00 0x7f67340f7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:26.263 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.273+0000 7f673a7df700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67340f53a0 0x7f67340f5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:26.263 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f673a7df700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67340f79c0 con 0x7f67340f53a0 2021-11-20T06:08:26.263 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f673a7df700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67340f5f60 con 0x7f67340f4a00 2021-11-20T06:08:26.264 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f67337fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67340f53a0 0x7f67340f5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:26.264 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f67337fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67340f53a0 0x7f67340f5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59260/0 (socket says 172.21.15.139:59260) 2021-11-20T06:08:26.264 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f67337fe700 1 -- 172.21.15.139:0/2676371219 learned_addr learned my addr 172.21.15.139:0/2676371219 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:26.264 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f6733fff700 1 --2- 172.21.15.139:0/2676371219 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67340f4a00 0x7f67340f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:26.265 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f67337fe700 1 -- 172.21.15.139:0/2676371219 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67340f4a00 msgr2=0x7f67340f7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:26.265 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f67337fe700 1 --2- 172.21.15.139:0/2676371219 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67340f4a00 0x7f67340f7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:26.265 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f67337fe700 1 -- 172.21.15.139:0/2676371219 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6728005040 con 0x7f67340f53a0 2021-11-20T06:08:26.265 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f6733fff700 1 --2- 172.21.15.139:0/2676371219 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67340f4a00 0x7f67340f7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:08:26.265 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f67337fe700 1 --2- 172.21.15.139:0/2676371219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67340f53a0 0x7f67340f5a70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f6728009f50 tx=0x7f6728009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:26.266 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f6730ff9700 1 -- 172.21.15.139:0/2676371219 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6728018070 con 0x7f67340f53a0 2021-11-20T06:08:26.266 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f6730ff9700 1 -- 172.21.15.139:0/2676371219 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6728013410 con 0x7f67340f53a0 2021-11-20T06:08:26.266 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f673a7df700 1 -- 172.21.15.139:0/2676371219 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67340f6180 con 0x7f67340f53a0 2021-11-20T06:08:26.266 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f6730ff9700 1 -- 172.21.15.139:0/2676371219 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f672801c620 con 0x7f67340f53a0 2021-11-20T06:08:26.267 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.274+0000 7f673a7df700 1 -- 172.21.15.139:0/2676371219 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67340f66e0 con 0x7f67340f53a0 2021-11-20T06:08:26.267 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.275+0000 7f673a7df700 1 -- 172.21.15.139:0/2676371219 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6734046330 con 0x7f67340f53a0 2021-11-20T06:08:26.268 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.276+0000 7f6730ff9700 1 -- 172.21.15.139:0/2676371219 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f672801d420 con 0x7f67340f53a0 2021-11-20T06:08:26.268 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.276+0000 7f6730ff9700 1 --2- 172.21.15.139:0/2676371219 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f671c05c2a0 0x7f671c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.276+0000 7f6730ff9700 1 -- 172.21.15.139:0/2676371219 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f6728078330 con 0x7f67340f53a0 2021-11-20T06:08:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.276+0000 7f6733fff700 1 --2- 172.21.15.139:0/2676371219 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f671c05c2a0 0x7f671c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.276+0000 7f6733fff700 1 --2- 172.21.15.139:0/2676371219 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f671c05c2a0 0x7f671c05e760 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f67240041d0 tx=0x7f6724009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:26.269 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.280+0000 7f6730ff9700 1 -- 172.21.15.139:0/2676371219 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f672804aa40 con 0x7f67340f53a0 2021-11-20T06:08:26.454 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.470+0000 7f673a7df700 1 -- 172.21.15.139:0/2676371219 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6734027a00 con 0x7f67340f53a0 2021-11-20T06:08:26.455 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.470+0000 7f6730ff9700 1 -- 172.21.15.139:0/2676371219 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+2184 (secure 0 0 0) 0x7f672804e060 con 0x7f67340f53a0 2021-11-20T06:08:26.455 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:26.456 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24615,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6827/339439062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":339439062},{"type":"v1","addr":"172.21.15.178:6827","nonce":339439062}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:19.258180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24583},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":88,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:26.458 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.472+0000 7f673a7df700 1 -- 172.21.15.139:0/2676371219 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f671c05c2a0 msgr2=0x7f671c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:26.458 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.472+0000 7f673a7df700 1 --2- 172.21.15.139:0/2676371219 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f671c05c2a0 0x7f671c05e760 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f67240041d0 tx=0x7f6724009040).stop 2021-11-20T06:08:26.459 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.472+0000 7f673a7df700 1 -- 172.21.15.139:0/2676371219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67340f53a0 msgr2=0x7f67340f5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:26.459 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.472+0000 7f673a7df700 1 --2- 172.21.15.139:0/2676371219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67340f53a0 0x7f67340f5a70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f6728009f50 tx=0x7f6728009b30).stop 2021-11-20T06:08:26.459 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.472+0000 7f673a7df700 1 -- 172.21.15.139:0/2676371219 shutdown_connections 2021-11-20T06:08:26.459 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.472+0000 7f673a7df700 1 --2- 172.21.15.139:0/2676371219 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f671c05c2a0 0x7f671c05e760 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:26.460 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.472+0000 7f673a7df700 1 --2- 172.21.15.139:0/2676371219 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67340f4a00 0x7f67340f7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:26.460 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.472+0000 7f673a7df700 1 --2- 172.21.15.139:0/2676371219 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f67340f53a0 0x7f67340f5a70 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:26.460 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.472+0000 7f673a7df700 1 -- 172.21.15.139:0/2676371219 >> 172.21.15.139:0/2676371219 conn(0x7f67340f0440 msgr2=0x7f67340fb730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:26.460 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.472+0000 7f673a7df700 1 -- 172.21.15.139:0/2676371219 shutdown_connections 2021-11-20T06:08:26.461 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:26.472+0000 7f673a7df700 1 -- 172.21.15.139:0/2676371219 wait complete. 2021-11-20T06:08:26.461 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 91 2021-11-20T06:08:26.469 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:19.258180+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24583}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 88, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/4081910640', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 4081910640}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 4081910640}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:26.469 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T06:08:27.471 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:27.592 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.605+0000 7f6a23fff700 1 -- 172.21.15.139:0/3134635690 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a240f65d0 msgr2=0x7f6a240f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:27.592 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.605+0000 7f6a23fff700 1 --2- 172.21.15.139:0/3134635690 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a240f65d0 0x7f6a240f69f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f6a14004660 tx=0x7f6a14009b30).stop 2021-11-20T06:08:27.593 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.605+0000 7f6a23fff700 1 -- 172.21.15.139:0/3134635690 shutdown_connections 2021-11-20T06:08:27.593 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.605+0000 7f6a23fff700 1 --2- 172.21.15.139:0/3134635690 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a240f7870 0x7f6a240f7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:27.593 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.605+0000 7f6a23fff700 1 --2- 172.21.15.139:0/3134635690 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a240f65d0 0x7f6a240f69f0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:27.593 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.605+0000 7f6a23fff700 1 -- 172.21.15.139:0/3134635690 >> 172.21.15.139:0/3134635690 conn(0x7f6a240f1d10 msgr2=0x7f6a240f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:27.594 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.605+0000 7f6a23fff700 1 -- 172.21.15.139:0/3134635690 shutdown_connections 2021-11-20T06:08:27.594 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.605+0000 7f6a23fff700 1 -- 172.21.15.139:0/3134635690 wait complete. 2021-11-20T06:08:27.594 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.606+0000 7f6a23fff700 1 Processor -- start 2021-11-20T06:08:27.595 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.606+0000 7f6a23fff700 1 -- start start 2021-11-20T06:08:27.595 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.606+0000 7f6a23fff700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a240f65d0 0x7f6a24106da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:27.595 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.606+0000 7f6a23fff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a240f7870 0x7f6a24107290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:27.595 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.606+0000 7f6a23fff700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a24107890 con 0x7f6a240f7870 2021-11-20T06:08:27.596 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.606+0000 7f6a23fff700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a241080d0 con 0x7f6a240f65d0 2021-11-20T06:08:27.596 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.606+0000 7f6a227fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a240f7870 0x7f6a24107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:27.596 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.606+0000 7f6a227fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a240f7870 0x7f6a24107290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59268/0 (socket says 172.21.15.139:59268) 2021-11-20T06:08:27.596 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.606+0000 7f6a227fc700 1 -- 172.21.15.139:0/269548333 learned_addr learned my addr 172.21.15.139:0/269548333 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:27.597 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.606+0000 7f6a22ffd700 1 --2- 172.21.15.139:0/269548333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a240f65d0 0x7f6a24106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:27.597 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.606+0000 7f6a227fc700 1 -- 172.21.15.139:0/269548333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a240f65d0 msgr2=0x7f6a24106da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:27.597 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.606+0000 7f6a227fc700 1 --2- 172.21.15.139:0/269548333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a240f65d0 0x7f6a24106da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:27.598 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.606+0000 7f6a227fc700 1 -- 172.21.15.139:0/269548333 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a14005040 con 0x7f6a240f7870 2021-11-20T06:08:27.598 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.607+0000 7f6a227fc700 1 --2- 172.21.15.139:0/269548333 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a240f7870 0x7f6a24107290 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f6a1800c790 tx=0x7f6a18007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:27.598 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.607+0000 7f6a0b7fe700 1 -- 172.21.15.139:0/269548333 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a1800e680 con 0x7f6a240f7870 2021-11-20T06:08:27.598 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.607+0000 7f6a0b7fe700 1 -- 172.21.15.139:0/269548333 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a1800ec90 con 0x7f6a240f7870 2021-11-20T06:08:27.599 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.607+0000 7f6a23fff700 1 -- 172.21.15.139:0/269548333 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a241083b0 con 0x7f6a240f7870 2021-11-20T06:08:27.599 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.607+0000 7f6a0b7fe700 1 -- 172.21.15.139:0/269548333 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a18013d60 con 0x7f6a240f7870 2021-11-20T06:08:27.599 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.607+0000 7f6a23fff700 1 -- 172.21.15.139:0/269548333 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a241088f0 con 0x7f6a240f7870 2021-11-20T06:08:27.599 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.608+0000 7f6a23fff700 1 -- 172.21.15.139:0/269548333 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a240f59c0 con 0x7f6a240f7870 2021-11-20T06:08:27.600 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.608+0000 7f6a0b7fe700 1 -- 172.21.15.139:0/269548333 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6a1801a060 con 0x7f6a240f7870 2021-11-20T06:08:27.601 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.608+0000 7f6a0b7fe700 1 --2- 172.21.15.139:0/269548333 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6a0c05c300 0x7f6a0c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:27.601 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.608+0000 7f6a0b7fe700 1 -- 172.21.15.139:0/269548333 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f6a18076690 con 0x7f6a240f7870 2021-11-20T06:08:27.601 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.608+0000 7f6a22ffd700 1 --2- 172.21.15.139:0/269548333 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6a0c05c300 0x7f6a0c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:27.601 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.609+0000 7f6a22ffd700 1 --2- 172.21.15.139:0/269548333 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6a0c05c300 0x7f6a0c05e7c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f6a24107b60 tx=0x7f6a14009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:27.602 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.612+0000 7f6a0b7fe700 1 -- 172.21.15.139:0/269548333 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a18048d40 con 0x7f6a240f7870 2021-11-20T06:08:27.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:27 smithi178 conmon[36600]: cluster 2021-11-20T06:08:26.284150+0000 mgr.smithi139.dplyxn (mgr.14162) 339 2021-11-20T06:08:27.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:27 smithi178 conmon[36600]: : cluster [DBG] pgmap v317: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:08:27.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:27 smithi178 conmon[36600]: audit 2021-11-20T06:08:26.471057+0000 mon.smithi139 (mon.0) 906 : audit [DBG] from='client.? 172.21.15.139:0/2676371219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:27.776 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:27 smithi139 conmon[33501]: cluster 2021-11-20T06:08:26.284150+0000 mgr.smithi139.dplyxn (mgr.14162) 339 : cluster [DBG] pgmap v317: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-20T06:08:27.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:27 smithi139 conmon[33501]: 2021-11-20T06:08:27.777 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:27 smithi139 conmon[33501]: audit 2021-11-20T06:08:26.471057+0000 mon.smithi139 (mon.0) 906 : audit [DBG] from='client.? 172.21.15.139:0/2676371219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:27.782 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.798+0000 7f6a23fff700 1 -- 172.21.15.139:0/269548333 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a240fee20 con 0x7f6a240f7870 2021-11-20T06:08:27.783 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.799+0000 7f6a0b7fe700 1 -- 172.21.15.139:0/269548333 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+2184 (secure 0 0 0) 0x7f6a1804c360 con 0x7f6a240f7870 2021-11-20T06:08:27.783 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:27.784 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24615,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6827/339439062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":339439062},{"type":"v1","addr":"172.21.15.178:6827","nonce":339439062}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:19.258180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24583},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":88,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:27.785 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.800+0000 7f6a23fff700 1 -- 172.21.15.139:0/269548333 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6a0c05c300 msgr2=0x7f6a0c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:27.785 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.800+0000 7f6a23fff700 1 --2- 172.21.15.139:0/269548333 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6a0c05c300 0x7f6a0c05e7c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f6a24107b60 tx=0x7f6a14009b30).stop 2021-11-20T06:08:27.785 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.800+0000 7f6a23fff700 1 -- 172.21.15.139:0/269548333 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a240f7870 msgr2=0x7f6a24107290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:27.786 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.800+0000 7f6a23fff700 1 --2- 172.21.15.139:0/269548333 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a240f7870 0x7f6a24107290 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f6a1800c790 tx=0x7f6a18007d70).stop 2021-11-20T06:08:27.786 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.800+0000 7f6a23fff700 1 -- 172.21.15.139:0/269548333 shutdown_connections 2021-11-20T06:08:27.786 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.800+0000 7f6a23fff700 1 --2- 172.21.15.139:0/269548333 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7f6a0c05c300 0x7f6a0c05e7c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:27.786 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.800+0000 7f6a23fff700 1 --2- 172.21.15.139:0/269548333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a240f65d0 0x7f6a24106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:27.787 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.800+0000 7f6a23fff700 1 --2- 172.21.15.139:0/269548333 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6a240f7870 0x7f6a24107290 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:27.787 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.800+0000 7f6a23fff700 1 -- 172.21.15.139:0/269548333 >> 172.21.15.139:0/269548333 conn(0x7f6a240f1d10 msgr2=0x7f6a240fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:27.787 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.800+0000 7f6a23fff700 1 -- 172.21.15.139:0/269548333 shutdown_connections 2021-11-20T06:08:27.787 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:27.800+0000 7f6a23fff700 1 -- 172.21.15.139:0/269548333 wait complete. 2021-11-20T06:08:27.787 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 91 2021-11-20T06:08:27.796 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:19.258180+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24583}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 88, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/4081910640', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 4081910640}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 4081910640}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:27.797 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T06:08:28.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:28 smithi178 conmon[36600]: audit 2021-11-20T06:08:27.799420+0000 mon.smithi139 (mon. 2021-11-20T06:08:28.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:28 smithi178 conmon[36600]: 0) 907 : audit [DBG] from='client.? 172.21.15.139:0/269548333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:28.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:28 smithi139 conmon[33501]: audit 2021-11-20T06:08:27.799420+0000 mon.smithi139 (mon.0 2021-11-20T06:08:28.778 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:28 smithi139 conmon[33501]: ) 907 : audit [DBG] from='client.? 172.21.15.139:0/269548333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:28.798 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-20T06:08:28.921 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.934+0000 7fc9bb62a700 1 -- 172.21.15.139:0/2635963393 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9b40f6820 msgr2=0x7fc9b40f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:28.922 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.934+0000 7fc9bb62a700 1 --2- 172.21.15.139:0/2635963393 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9b40f6820 0x7fc9b40f6c80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7fc9a4004660 tx=0x7fc9a4009b30).stop 2021-11-20T06:08:28.922 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.934+0000 7fc9bb62a700 1 -- 172.21.15.139:0/2635963393 shutdown_connections 2021-11-20T06:08:28.922 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.934+0000 7fc9bb62a700 1 --2- 172.21.15.139:0/2635963393 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9b40f6820 0x7fc9b40f6c80 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:28.923 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.934+0000 7fc9bb62a700 1 --2- 172.21.15.139:0/2635963393 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9b40f5580 0x7fc9b40f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:28.923 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.934+0000 7fc9bb62a700 1 -- 172.21.15.139:0/2635963393 >> 172.21.15.139:0/2635963393 conn(0x7fc9b40f0ce0 msgr2=0x7fc9b40f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:28.923 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.935+0000 7fc9bb62a700 1 -- 172.21.15.139:0/2635963393 shutdown_connections 2021-11-20T06:08:28.923 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.935+0000 7fc9bb62a700 1 -- 172.21.15.139:0/2635963393 wait complete. 2021-11-20T06:08:28.924 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.935+0000 7fc9bb62a700 1 Processor -- start 2021-11-20T06:08:28.924 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.935+0000 7fc9bb62a700 1 -- start start 2021-11-20T06:08:28.925 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.935+0000 7fc9bb62a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9b40f5580 0x7fc9b4105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:28.925 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.935+0000 7fc9bb62a700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9b40f6820 0x7fc9b4106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:28.925 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.935+0000 7fc9bb62a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b4106890 con 0x7fc9b40f5580 2021-11-20T06:08:28.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.935+0000 7fc9bb62a700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b41069d0 con 0x7fc9b40f6820 2021-11-20T06:08:28.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.935+0000 7fc9b93c6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9b40f5580 0x7fc9b4105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:28.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.935+0000 7fc9b93c6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9b40f5580 0x7fc9b4105da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:59278/0 (socket says 172.21.15.139:59278) 2021-11-20T06:08:28.926 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.935+0000 7fc9b93c6700 1 -- 172.21.15.139:0/3329500424 learned_addr learned my addr 172.21.15.139:0/3329500424 (peer_addr_for_me v2:172.21.15.139:0/0) 2021-11-20T06:08:28.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.935+0000 7fc9b8bc5700 1 --2- 172.21.15.139:0/3329500424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9b40f6820 0x7fc9b4106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:28.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.935+0000 7fc9b93c6700 1 -- 172.21.15.139:0/3329500424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9b40f6820 msgr2=0x7fc9b4106290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:28.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.936+0000 7fc9b93c6700 1 --2- 172.21.15.139:0/3329500424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9b40f6820 0x7fc9b4106290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:28.927 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.936+0000 7fc9b93c6700 1 -- 172.21.15.139:0/3329500424 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a4005040 con 0x7fc9b40f5580 2021-11-20T06:08:28.928 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.936+0000 7fc9b8bc5700 1 --2- 172.21.15.139:0/3329500424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9b40f6820 0x7fc9b4106290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-20T06:08:28.928 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.936+0000 7fc9b93c6700 1 --2- 172.21.15.139:0/3329500424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9b40f5580 0x7fc9b4105da0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fc9b0002700 tx=0x7fc9b0009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:28.928 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.936+0000 7fc9a9ffb700 1 -- 172.21.15.139:0/3329500424 <== mon.0 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9b0009500 con 0x7fc9b40f5580 2021-11-20T06:08:28.928 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.936+0000 7fc9bb62a700 1 -- 172.21.15.139:0/3329500424 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b405fa60 con 0x7fc9b40f5580 2021-11-20T06:08:28.929 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.936+0000 7fc9a9ffb700 1 -- 172.21.15.139:0/3329500424 <== mon.0 v2:172.21.15.139:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9b0009660 con 0x7fc9b40f5580 2021-11-20T06:08:28.929 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.936+0000 7fc9a9ffb700 1 -- 172.21.15.139:0/3329500424 <== mon.0 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9b0011c70 con 0x7fc9b40f5580 2021-11-20T06:08:28.929 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.936+0000 7fc9bb62a700 1 -- 172.21.15.139:0/3329500424 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b4060020 con 0x7fc9b40f5580 2021-11-20T06:08:28.930 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.937+0000 7fc9bb62a700 1 -- 172.21.15.139:0/3329500424 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b40fdd10 con 0x7fc9b40f5580 2021-11-20T06:08:28.930 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.937+0000 7fc9a9ffb700 1 -- 172.21.15.139:0/3329500424 <== mon.0 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc9b00077b0 con 0x7fc9b40f5580 2021-11-20T06:08:28.930 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.938+0000 7fc9a9ffb700 1 --2- 172.21.15.139:0/3329500424 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc9a005c2a0 0x7fc9a005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-20T06:08:28.931 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.938+0000 7fc9a9ffb700 1 -- 172.21.15.139:0/3329500424 <== mon.0 v2:172.21.15.139:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7fc9b0078b10 con 0x7fc9b40f5580 2021-11-20T06:08:28.931 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.938+0000 7fc9b8bc5700 1 --2- 172.21.15.139:0/3329500424 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc9a005c2a0 0x7fc9a005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-20T06:08:28.932 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.938+0000 7fc9b8bc5700 1 --2- 172.21.15.139:0/3329500424 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc9a005c2a0 0x7fc9a005e760 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fc9a4009d60 tx=0x7fc9a4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-20T06:08:28.932 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:28.941+0000 7fc9a9ffb700 1 -- 172.21.15.139:0/3329500424 <== mon.0 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9b004b220 con 0x7fc9b40f5580 2021-11-20T06:08:29.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:29.126+0000 7fc9bb62a700 1 -- 172.21.15.139:0/3329500424 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9b410b380 con 0x7fc9b40f5580 2021-11-20T06:08:29.111 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:29.127+0000 7fc9a9ffb700 1 -- 172.21.15.139:0/3329500424 <== mon.0 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+2184 (secure 0 0 0) 0x7fc9b004e840 con 0x7fc9b40f5580 2021-11-20T06:08:29.112 INFO:teuthology.orchestra.run.smithi139.stdout: 2021-11-20T06:08:29.112 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24615,"name":"cephfs.smithi178.pskihy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.178:6827/339439062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":339439062},{"type":"v1","addr":"172.21.15.178:6827","nonce":339439062}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-20T06:03:25.199256+0000","modified":"2021-11-20T06:08:19.258180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24583},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24583":{"gid":24583,"name":"cephfs.smithi178.blvkbx","rank":0,"incarnation":88,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.178:6825/4081910640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":4081910640},{"type":"v1","addr":"172.21.15.178:6825","nonce":4081910640}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-20T06:08:29.113 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:29.129+0000 7fc9bb62a700 1 -- 172.21.15.139:0/3329500424 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc9a005c2a0 msgr2=0x7fc9a005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:29.114 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:29.129+0000 7fc9bb62a700 1 --2- 172.21.15.139:0/3329500424 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc9a005c2a0 0x7fc9a005e760 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fc9a4009d60 tx=0x7fc9a4009b30).stop 2021-11-20T06:08:29.114 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:29.129+0000 7fc9bb62a700 1 -- 172.21.15.139:0/3329500424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9b40f5580 msgr2=0x7fc9b4105da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-20T06:08:29.114 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:29.129+0000 7fc9bb62a700 1 --2- 172.21.15.139:0/3329500424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9b40f5580 0x7fc9b4105da0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fc9b0002700 tx=0x7fc9b0009d80).stop 2021-11-20T06:08:29.115 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:29.129+0000 7fc9bb62a700 1 -- 172.21.15.139:0/3329500424 shutdown_connections 2021-11-20T06:08:29.115 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:29.129+0000 7fc9bb62a700 1 --2- 172.21.15.139:0/3329500424 >> [v2:172.21.15.139:6800/2363962993,v1:172.21.15.139:6801/2363962993] conn(0x7fc9a005c2a0 0x7fc9a005e760 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:29.115 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:29.129+0000 7fc9bb62a700 1 --2- 172.21.15.139:0/3329500424 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9b40f5580 0x7fc9b4105da0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:29.115 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:29.129+0000 7fc9bb62a700 1 --2- 172.21.15.139:0/3329500424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9b40f6820 0x7fc9b4106290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-20T06:08:29.115 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:29.129+0000 7fc9bb62a700 1 -- 172.21.15.139:0/3329500424 >> 172.21.15.139:0/3329500424 conn(0x7fc9b40f0ce0 msgr2=0x7fc9b40fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-20T06:08:29.116 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:29.129+0000 7fc9bb62a700 1 -- 172.21.15.139:0/3329500424 shutdown_connections 2021-11-20T06:08:29.116 INFO:teuthology.orchestra.run.smithi139.stderr:2021-11-20T06:08:29.129+0000 7fc9bb62a700 1 -- 172.21.15.139:0/3329500424 wait complete. 2021-11-20T06:08:29.116 INFO:teuthology.orchestra.run.smithi139.stderr:dumped fsmap epoch 91 2021-11-20T06:08:29.125 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-20T06:03:25.199256+0000', 'modified': '2021-11-20T06:08:19.258180+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24583}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24583': {'gid': 24583, 'name': 'cephfs.smithi178.blvkbx', 'rank': 0, 'incarnation': 88, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.178:6825/4081910640', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6824', 'nonce': 4081910640}, {'type': 'v1', 'addr': '172.21.15.178:6825', 'nonce': 4081910640}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-20T06:08:29.126 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24583:up:rejoin 2021-11-20T06:08:29.526 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:29 smithi139 conmon[33501]: cluster 2021-11-20T06:08:28.284690+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:08:29.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:29 smithi139 conmon[33501]: ) 340 : cluster [DBG] pgmap v318: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:08:29.527 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:29 smithi139 conmon[33501]: audit 2021-11-20T06:08:29.127853+0000 mon.smithi139 (mon.0) 908 : audit [DBG] from='client.? 172.21.15.139:0/3329500424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:29.691 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:29 smithi178 conmon[36600]: cluster 2021-11-20T06:08:28.284690+0000 mgr.smithi139.dplyxn (mgr.14162 2021-11-20T06:08:29.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:29 smithi178 conmon[36600]: ) 340 : cluster [DBG] pgmap v318: 65 pgs: 65 active+clean; 2.5 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-20T06:08:29.692 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:29 smithi178 conmon[36600]: audit 2021-11-20T06:08:29.127853+0000 mon.smithi139 (mon.0) 908 : audit [DBG] from='client.? 172.21.15.139:0/3329500424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-20T06:08:30.127 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 91, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-11-20T06:03:25.199256+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 90, "ever_allowed_features": 32, "explicitly_allowed_features": 32, "failed": [], "flags": 50, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_24583": { "addr": "172.21.15.178:6825/4081910640", "addrs": { "addrvec": [ { "addr": "172.21.15.178:6824", "nonce": 4081910640, "type": "v2" }, { "addr": "172.21.15.178:6825", "nonce": 4081910640, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 24583, "incarnation": 88, "join_fscid": 1, "name": "cephfs.smithi178.blvkbx", "rank": 0, "state": "up:rejoin", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "modified": "2021-11-20T06:08:19.258180+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 1, "stopped": [], "tableserver": 0, "up": { "mds_0": 24583 } } } ], "standbys": [ { "addr": "172.21.15.178:6827/339439062", "addrs": { "addrvec": [ { "addr": "172.21.15.178:6826", "nonce": 339439062, "type": "v2" }, { "addr": "172.21.15.178:6827", "nonce": 339439062, "type": "v1" } ] }, "epoch": 91, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 24615, "incarnation": 0, "join_fscid": 1, "name": "cephfs.smithi178.pskihy", "rank": -1, "state": "up:standby", "state_seq": 3 } ] } 2021-11-20T06:08:30.128 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/kclient.py", line 112, in task kernel_mount.mount(mntopts=client_config.get('mntopts', [])) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/kernel_mount.py", line 44, in mount self.setupfs(name=self.cephfs_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/mount.py", line 161, in setupfs self.fs.wait_for_daemons() File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/filesystem.py", line 1109, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2021-11-20T06:08:30.267 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=66b343e473634de28f54e7fd66435a9a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/kclient.py", line 112, in task kernel_mount.mount(mntopts=client_config.get('mntopts', [])) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/kernel_mount.py", line 44, in mount self.setupfs(name=self.cephfs_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/mount.py", line 161, in setupfs self.fs.wait_for_daemons() File "/home/teuthworker/src/github.com_ceph_ceph-c_9a97ad1bc894bb05b5a58acd3c053b0288a9f4aa/qa/tasks/cephfs/filesystem.py", line 1109, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2021-11-20T06:08:30.272 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2021-11-20T06:08:30.286 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2021-11-20T06:08:30.424 INFO:tasks.cephadm:Teardown begin 2021-11-20T06:08:30.425 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T06:08:30.440 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:30 smithi178 conmon[36600]: cluster 2021-11-20T06:08:30.204733+0000 mon.smithi139 (mon.0) 909 : cluster [INF] daemon mds.cephfs.smithi178.blvkbx restarted 2021-11-20T06:08:30.441 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:30 smithi178 conmon[36600]: cluster 2021-11-20T06:08:30.209404+0000 mon.smithi139 (mon.0) 910 : cluster [DBG] osdmap e59: 6 total, 6 up, 6 in 2021-11-20T06:08:30.461 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T06:08:30.502 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2021-11-20T06:08:30.503 DEBUG:teuthology.orchestra.run.smithi139:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2021-11-20T06:08:30.521 DEBUG:teuthology.orchestra.run.smithi178:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2021-11-20T06:08:30.559 INFO:tasks.cephadm:Stopping all daemons... 2021-11-20T06:08:30.560 INFO:tasks.cephadm.mon.smithi139:Stopping mon.smithi139... 2021-11-20T06:08:30.560 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi139 2021-11-20T06:08:30.587 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:30 smithi139 conmon[33501]: debug 2021-11-20T06:08:30.342+0000 7f2a21654700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-20T06:08:30.587 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:30 smithi139 conmon[33501]: cluster 2021-11-20T06:08: 2021-11-20T06:08:30.588 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:30 smithi139 conmon[33501]: 30.204733+0000 mon.smithi139 (mon.0) 909 : cluster [INF] daemon mds.cephfs.smithi178.blvkbx restarted 2021-11-20T06:08:30.589 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:30 smithi139 conmon[33501]: cluster 2021-11-20T06:08:30 2021-11-20T06:08:30.589 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:30 smithi139 conmon[33501]: .209404+0000 mon.smithi139 (mon.0) 910 : cluster [DBG] osdmap e59: 6 total, 6 up, 6 in 2021-11-20T06:08:30.847 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:30 smithi139 systemd[1]: Stopping Ceph mon.smithi139 for d819f2cc-49c6-11ec-8c2c-001a4aab830c... 2021-11-20T06:08:30.847 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:30 smithi139 conmon[33501]: debug 2021-11-20T06:08:30.738+0000 7f2a2565c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi139 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-11-20T06:08:30.848 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:30 smithi139 conmon[33501]: debug 2021-11-20T06:08:30.738+0000 7f2a2565c700 -1 mon.smithi139@0(leader) e2 *** Got Signal Terminated *** 2021-11-20T06:08:31.090 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi139.service' 2021-11-20T06:08:31.155 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:31 smithi139 podman[96028]: ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c-mon.smithi139 2021-11-20T06:08:31.156 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:31 smithi139 systemd[1]: ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi139.service: Succeeded. 2021-11-20T06:08:31.156 INFO:journalctl@ceph.mon.smithi139.smithi139.stdout:Nov 20 06:08:31 smithi139 systemd[1]: Stopped Ceph mon.smithi139 for d819f2cc-49c6-11ec-8c2c-001a4aab830c. 2021-11-20T06:08:31.180 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T06:08:31.181 INFO:tasks.cephadm.mon.smithi139:Stopped mon.smithi139 2021-11-20T06:08:31.181 INFO:tasks.cephadm.mon.smithi178:Stopping mon.smithi178... 2021-11-20T06:08:31.182 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi178 2021-11-20T06:08:31.488 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:31 smithi178 systemd[1]: Stopping Ceph mon.smithi178 for d819f2cc-49c6-11ec-8c2c-001a4aab830c... 2021-11-20T06:08:31.489 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:31 smithi178 conmon[36600]: debug 2021-11-20T06:08:31.308+0000 7fb4f1afd700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi178 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-11-20T06:08:31.489 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:31 smithi178 conmon[36600]: debug 2021-11-20T06:08:31.308+0000 7fb4f1afd700 -1 mon.smithi178@1(peon) e2 *** Got Signal Terminated *** 2021-11-20T06:08:31.749 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:Nov 20 06:08:31 smithi178 podman[60027]: ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c-mon.smithi178 2021-11-20T06:08:31.757 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@mon.smithi178.service' 2021-11-20T06:08:31.807 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T06:08:31.807 INFO:tasks.cephadm.mon.smithi178:Stopped mon.smithi178 2021-11-20T06:08:31.808 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2021-11-20T06:08:31.808 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.0 2021-11-20T06:08:31.929 INFO:journalctl@ceph.osd.0.smithi139.stdout:Nov 20 06:08:31 smithi139 systemd[1]: Stopping Ceph osd.0 for d819f2cc-49c6-11ec-8c2c-001a4aab830c... 2021-11-20T06:08:32.188 INFO:journalctl@ceph.osd.0.smithi139.stdout:Nov 20 06:08:31 smithi139 conmon[50551]: debug 2021-11-20T06:08:31.945+0000 7f1f181ee700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T06:08:32.189 INFO:journalctl@ceph.osd.0.smithi139.stdout:Nov 20 06:08:31 smithi139 conmon[50551]: debug 2021-11-20T06:08:31.945+0000 7f1f181ee700 -1 osd.0 59 *** Got signal Terminated *** 2021-11-20T06:08:32.189 INFO:journalctl@ceph.osd.0.smithi139.stdout:Nov 20 06:08:31 smithi139 conmon[50551]: debug 2021-11-20T06:08:31.945+0000 7f1f181ee700 -1 osd.0 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T06:08:32.509 INFO:journalctl@ceph.osd.0.smithi139.stdout:Nov 20 06:08:32 smithi139 podman[96155]: ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c-osd.0 2021-11-20T06:08:32.776 INFO:journalctl@ceph.osd.0.smithi139.stdout:Nov 20 06:08:32 smithi139 podman[96319]: 2021-11-20T06:08:33.299 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.0.service' 2021-11-20T06:08:33.339 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T06:08:33.340 INFO:tasks.cephadm.osd.0:Stopped osd.0 2021-11-20T06:08:33.340 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2021-11-20T06:08:33.340 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.1 2021-11-20T06:08:33.621 INFO:journalctl@ceph.osd.1.smithi139.stdout:Nov 20 06:08:33 smithi139 systemd[1]: Stopping Ceph osd.1 for d819f2cc-49c6-11ec-8c2c-001a4aab830c... 2021-11-20T06:08:33.621 INFO:journalctl@ceph.osd.1.smithi139.stdout:Nov 20 06:08:33 smithi139 conmon[55514]: debug 2021-11-20T06:08:33.498+0000 7f43fe9b6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T06:08:33.622 INFO:journalctl@ceph.osd.1.smithi139.stdout:Nov 20 06:08:33 smithi139 conmon[55514]: debug 2021-11-20T06:08:33.498+0000 7f43fe9b6700 -1 osd.1 59 *** Got signal Terminated *** 2021-11-20T06:08:33.622 INFO:journalctl@ceph.osd.1.smithi139.stdout:Nov 20 06:08:33 smithi139 conmon[55514]: debug 2021-11-20T06:08:33.498+0000 7f43fe9b6700 -1 osd.1 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T06:08:34.026 INFO:journalctl@ceph.osd.1.smithi139.stdout:Nov 20 06:08:33 smithi139 podman[96480]: ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c-osd.1 2021-11-20T06:08:34.513 INFO:journalctl@ceph.osd.1.smithi139.stdout:Nov 20 06:08:34 smithi139 podman[96643]: 2021-11-20T06:08:34.866 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.1.service' 2021-11-20T06:08:34.904 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T06:08:34.905 INFO:tasks.cephadm.osd.1:Stopped osd.1 2021-11-20T06:08:34.905 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2021-11-20T06:08:34.906 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.2 2021-11-20T06:08:35.187 INFO:journalctl@ceph.osd.2.smithi139.stdout:Nov 20 06:08:34 smithi139 systemd[1]: Stopping Ceph osd.2 for d819f2cc-49c6-11ec-8c2c-001a4aab830c... 2021-11-20T06:08:35.188 INFO:journalctl@ceph.osd.2.smithi139.stdout:Nov 20 06:08:35 smithi139 conmon[61885]: debug 2021-11-20T06:08:35.073+0000 7f0a5beda700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T06:08:35.188 INFO:journalctl@ceph.osd.2.smithi139.stdout:Nov 20 06:08:35 smithi139 conmon[61885]: debug 2021-11-20T06:08:35.073+0000 7f0a5beda700 -1 osd.2 59 *** Got signal Terminated *** 2021-11-20T06:08:35.189 INFO:journalctl@ceph.osd.2.smithi139.stdout:Nov 20 06:08:35 smithi139 conmon[61885]: debug 2021-11-20T06:08:35.073+0000 7f0a5beda700 -1 osd.2 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T06:08:35.527 INFO:journalctl@ceph.osd.2.smithi139.stdout:Nov 20 06:08:35 smithi139 podman[96803]: ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c-osd.2 2021-11-20T06:08:36.026 INFO:journalctl@ceph.osd.2.smithi139.stdout:Nov 20 06:08:35 smithi139 podman[96969]: 2021-11-20T06:08:36.531 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.2.service' 2021-11-20T06:08:36.569 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T06:08:36.570 INFO:tasks.cephadm.osd.2:Stopped osd.2 2021-11-20T06:08:36.571 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2021-11-20T06:08:36.571 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.3 2021-11-20T06:08:36.887 INFO:journalctl@ceph.osd.3.smithi178.stdout:Nov 20 06:08:36 smithi178 systemd[1]: Stopping Ceph osd.3 for d819f2cc-49c6-11ec-8c2c-001a4aab830c... 2021-11-20T06:08:36.888 INFO:journalctl@ceph.osd.3.smithi178.stdout:Nov 20 06:08:36 smithi178 conmon[43163]: debug 2021-11-20T06:08:36.727+0000 7fc60e8d2700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T06:08:36.888 INFO:journalctl@ceph.osd.3.smithi178.stdout:Nov 20 06:08:36 smithi178 conmon[43163]: debug 2021-11-20T06:08:36.727+0000 7fc60e8d2700 -1 osd.3 59 *** Got signal Terminated *** 2021-11-20T06:08:36.888 INFO:journalctl@ceph.osd.3.smithi178.stdout:Nov 20 06:08:36 smithi178 conmon[43163]: debug 2021-11-20T06:08:36.727+0000 7fc60e8d2700 -1 osd.3 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T06:08:37.191 INFO:journalctl@ceph.osd.3.smithi178.stdout:Nov 20 06:08:37 smithi178 podman[60163]: ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c-osd.3 2021-11-20T06:08:37.691 INFO:journalctl@ceph.osd.3.smithi178.stdout:Nov 20 06:08:37 smithi178 podman[60328]: 2021-11-20T06:08:38.070 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.3.service' 2021-11-20T06:08:38.109 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T06:08:38.109 INFO:tasks.cephadm.osd.3:Stopped osd.3 2021-11-20T06:08:38.110 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2021-11-20T06:08:38.110 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.4 2021-11-20T06:08:38.420 INFO:journalctl@ceph.osd.4.smithi178.stdout:Nov 20 06:08:38 smithi178 systemd[1]: Stopping Ceph osd.4 for d819f2cc-49c6-11ec-8c2c-001a4aab830c... 2021-11-20T06:08:38.420 INFO:journalctl@ceph.osd.4.smithi178.stdout:Nov 20 06:08:38 smithi178 conmon[48023]: debug 2021-11-20T06:08:38.286+0000 7f3e10901700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T06:08:38.421 INFO:journalctl@ceph.osd.4.smithi178.stdout:Nov 20 06:08:38 smithi178 conmon[48023]: debug 2021-11-20T06:08:38.286+0000 7f3e10901700 -1 osd.4 59 *** Got signal Terminated *** 2021-11-20T06:08:38.421 INFO:journalctl@ceph.osd.4.smithi178.stdout:Nov 20 06:08:38 smithi178 conmon[48023]: debug 2021-11-20T06:08:38.286+0000 7f3e10901700 -1 osd.4 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T06:08:38.690 INFO:journalctl@ceph.osd.4.smithi178.stdout:Nov 20 06:08:38 smithi178 podman[60485]: ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c-osd.4 2021-11-20T06:08:39.190 INFO:journalctl@ceph.osd.4.smithi178.stdout:Nov 20 06:08:38 smithi178 podman[60648]: 2021-11-20T06:08:39.630 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.4.service' 2021-11-20T06:08:39.668 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T06:08:39.669 INFO:tasks.cephadm.osd.4:Stopped osd.4 2021-11-20T06:08:39.670 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2021-11-20T06:08:39.670 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.5 2021-11-20T06:08:39.941 INFO:journalctl@ceph.osd.5.smithi178.stdout:Nov 20 06:08:39 smithi178 systemd[1]: Stopping Ceph osd.5 for d819f2cc-49c6-11ec-8c2c-001a4aab830c... 2021-11-20T06:08:39.941 INFO:journalctl@ceph.osd.5.smithi178.stdout:Nov 20 06:08:39 smithi178 conmon[52933]: debug 2021-11-20T06:08:39.852+0000 7fbf1310a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-20T06:08:39.942 INFO:journalctl@ceph.osd.5.smithi178.stdout:Nov 20 06:08:39 smithi178 conmon[52933]: debug 2021-11-20T06:08:39.852+0000 7fbf1310a700 -1 osd.5 59 *** Got signal Terminated *** 2021-11-20T06:08:39.942 INFO:journalctl@ceph.osd.5.smithi178.stdout:Nov 20 06:08:39 smithi178 conmon[52933]: debug 2021-11-20T06:08:39.852+0000 7fbf1310a700 -1 osd.5 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-20T06:08:40.375 INFO:journalctl@ceph.osd.5.smithi178.stdout:Nov 20 06:08:40 smithi178 podman[60808]: ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c-osd.5 2021-11-20T06:08:40.691 INFO:journalctl@ceph.osd.5.smithi178.stdout:Nov 20 06:08:40 smithi178 podman[60974]: 2021-11-20T06:08:41.194 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d819f2cc-49c6-11ec-8c2c-001a4aab830c@osd.5.service' 2021-11-20T06:08:41.232 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-20T06:08:41.232 INFO:tasks.cephadm.osd.5:Stopped osd.5 2021-11-20T06:08:41.233 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c --force --keep-logs 2021-11-20T06:08:45.838 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c --force --keep-logs 2021-11-20T06:08:55.109 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T06:08:55.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-20T06:08:55.178 INFO:tasks.cephadm:Archiving crash dumps... 2021-11-20T06:08:55.181 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514834/remote/ubuntu@smithi139.front.sepia.ceph.com/crash 2021-11-20T06:08:55.182 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar cz -f - -C /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/crash -- . 2021-11-20T06:08:55.235 INFO:teuthology.orchestra.run.smithi139.stderr:tar: /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/crash: Cannot open: No such file or directory 2021-11-20T06:08:55.235 INFO:teuthology.orchestra.run.smithi139.stderr:tar: Error is not recoverable: exiting now 2021-11-20T06:08:55.237 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514834/remote/ubuntu@smithi178.front.sepia.ceph.com/crash 2021-11-20T06:08:55.239 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar cz -f - -C /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/crash -- . 2021-11-20T06:08:55.276 INFO:teuthology.orchestra.run.smithi178.stderr:tar: /var/lib/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/crash: Cannot open: No such file or directory 2021-11-20T06:08:55.277 INFO:teuthology.orchestra.run.smithi178.stderr:tar: Error is not recoverable: exiting now 2021-11-20T06:08:55.278 INFO:tasks.cephadm:Checking cluster log for badness... 2021-11-20T06:08:55.279 DEBUG:teuthology.orchestra.run.smithi139:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2021-11-20T06:08:55.310 INFO:teuthology.orchestra.run.smithi139.stderr:grep: /var/log/ceph/d819f2cc-49c6-11ec-8c2c-001a4aab830c/ceph.log: No such file or directory 2021-11-20T06:08:55.313 INFO:tasks.cephadm:Compressing logs... 2021-11-20T06:08:55.314 DEBUG:teuthology.orchestra.run.smithi139:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T06:08:55.356 DEBUG:teuthology.orchestra.run.smithi178:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T06:08:55.381 INFO:teuthology.orchestra.run.smithi139.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2021-11-20T06:08:55.382 INFO:teuthology.orchestra.run.smithi178.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2021-11-20T06:08:56.445 INFO:tasks.cephadm:Archiving logs... 2021-11-20T06:08:56.446 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/log/ceph to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514834/remote/ubuntu@smithi139.front.sepia.ceph.com/log 2021-11-20T06:08:56.448 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar cz -f - -C /var/log/ceph -- . 2021-11-20T06:08:56.749 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/log/ceph to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514834/remote/ubuntu@smithi178.front.sepia.ceph.com/log 2021-11-20T06:08:56.750 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar cz -f - -C /var/log/ceph -- . 2021-11-20T06:08:56.924 INFO:tasks.cephadm:Removing cluster... 2021-11-20T06:08:56.924 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c --force 2021-11-20T06:08:57.432 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d819f2cc-49c6-11ec-8c2c-001a4aab830c --force 2021-11-20T06:08:57.940 INFO:tasks.cephadm:Removing cephadm ... 2021-11-20T06:08:57.941 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf /home/ubuntu/cephtest/cephadm 2021-11-20T06:08:57.957 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf /home/ubuntu/cephtest/cephadm 2021-11-20T06:08:57.973 INFO:tasks.cephadm:Teardown complete 2021-11-20T06:08:57.973 DEBUG:teuthology.run_tasks:Unwinding manager install 2021-11-20T06:08:57.986 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2021-11-20T06:08:57.987 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-11-20T06:08:58.000 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-11-20T06:08:58.086 DEBUG:teuthology.run_tasks:Unwinding manager clock 2021-11-20T06:08:58.113 INFO:teuthology.task.clock:Checking final clock skew... 2021-11-20T06:08:58.114 DEBUG:teuthology.orchestra.run.smithi139:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T06:08:58.117 DEBUG:teuthology.orchestra.run.smithi178:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-20T06:08:58.129 INFO:teuthology.orchestra.run.smithi139.stderr:bash: ntpq: command not found 2021-11-20T06:08:58.131 INFO:teuthology.orchestra.run.smithi178.stderr:bash: ntpq: command not found 2021-11-20T06:08:58.133 INFO:teuthology.orchestra.run.smithi139.stdout:210 Number of sources = 4 2021-11-20T06:08:58.133 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T06:08:58.134 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2021-11-20T06:08:58.134 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 44 -3182us[-3209us] +/- 94ms 2021-11-20T06:08:58.134 INFO:teuthology.orchestra.run.smithi139.stdout:^* hv02.front.sepia.ceph.com 2 6 377 41 +179us[ +152us] +/- 35ms 2021-11-20T06:08:58.135 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 +250us[ +223us] +/- 61ms 2021-11-20T06:08:58.135 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 41 +1696us[+1696us] +/- 62ms 2021-11-20T06:08:58.136 INFO:teuthology.orchestra.run.smithi178.stdout:210 Number of sources = 4 2021-11-20T06:08:58.136 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-20T06:08:58.137 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2021-11-20T06:08:58.137 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 40 -3184us[-3184us] +/- 95ms 2021-11-20T06:08:58.137 INFO:teuthology.orchestra.run.smithi178.stdout:^* hv02.front.sepia.ceph.com 2 6 377 42 +177us[ +196us] +/- 34ms 2021-11-20T06:08:58.138 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 +239us[ +239us] +/- 61ms 2021-11-20T06:08:58.138 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 40 +1678us[+1678us] +/- 62ms 2021-11-20T06:08:58.139 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-11-20T06:08:58.176 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-11-20T06:08:58.177 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-11-20T06:08:58.229 DEBUG:teuthology.orchestra.run.smithi139:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-11-20T06:08:58.291 DEBUG:teuthology.orchestra.run.smithi178:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-11-20T06:08:58.349 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T06:08:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T06:08:58.378 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-20T06:08:58.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T06:08:58.407 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-11-20T06:08:58.418 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-11-20T06:08:58.449 INFO:teuthology.task.internal:Duration was 1202.726608 seconds 2021-11-20T06:08:58.451 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-11-20T06:08:58.512 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-11-20T06:08:58.513 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-11-20T06:08:58.515 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-11-20T06:08:58.557 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T06:08:58.559 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-20T06:08:58.787 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2021-11-20T06:08:58.787 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi139.front.sepia.ceph.com 2021-11-20T06:08:58.788 DEBUG:teuthology.orchestra.run.smithi139:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-11-20T06:08:58.847 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com 2021-11-20T06:08:58.848 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-11-20T06:08:58.868 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2021-11-20T06:08:58.869 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T06:08:58.891 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-20T06:08:58.959 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2021-11-20T06:08:58.960 DEBUG:teuthology.orchestra.run.smithi139:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2021-11-20T06:08:58.962 DEBUG:teuthology.orchestra.run.smithi178:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2021-11-20T06:08:59.113 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-11-20T06:08:59.127 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-11-20T06:08:59.128 DEBUG:teuthology.orchestra.run.smithi139:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-11-20T06:08:59.153 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-11-20T06:08:59.178 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-11-20T06:08:59.218 DEBUG:teuthology.orchestra.run.smithi139:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-11-20T06:08:59.221 DEBUG:teuthology.orchestra.run.smithi178:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-11-20T06:08:59.243 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = core 2021-11-20T06:08:59.245 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = core 2021-11-20T06:08:59.259 DEBUG:teuthology.orchestra.run.smithi139:> test -e /home/ubuntu/cephtest/archive/coredump 2021-11-20T06:08:59.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T06:08:59.312 DEBUG:teuthology.orchestra.run.smithi178:> test -e /home/ubuntu/cephtest/archive/coredump 2021-11-20T06:08:59.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-20T06:08:59.328 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-11-20T06:08:59.343 INFO:teuthology.task.internal:Transferring archived files... 2021-11-20T06:08:59.344 DEBUG:teuthology.misc:Transferring archived files from smithi139:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514834/remote/smithi139 2021-11-20T06:08:59.345 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-11-20T06:08:59.412 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2021-11-20_04:32:48-rados-wip-yuri2-testing-2021-11-19-1714-pacific-distro-basic-smithi/6514834/remote/smithi178 2021-11-20T06:08:59.413 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-11-20T06:08:59.460 INFO:teuthology.task.internal:Removing archive directory... 2021-11-20T06:08:59.461 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf -- /home/ubuntu/cephtest/archive 2021-11-20T06:08:59.464 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf -- /home/ubuntu/cephtest/archive 2021-11-20T06:08:59.512 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-11-20T06:08:59.527 INFO:teuthology.task.internal:Not uploading archives. 2021-11-20T06:08:59.527 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-11-20T06:08:59.606 INFO:teuthology.task.internal:Tidying up after the test... 2021-11-20T06:08:59.606 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-11-20T06:08:59.609 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-11-20T06:08:59.624 INFO:teuthology.orchestra.run.smithi178.stdout: 262155 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 20 06:08 /home/ubuntu/cephtest 2021-11-20T15:34:08.156 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-11-20T15:34:08.190 DEBUG:teuthology.task.console_log:Killing console logger for smithi139 2021-11-20T15:34:08.191 DEBUG:teuthology.task.console_log:Killing console logger for smithi178 2021-11-20T15:34:08.192 DEBUG:teuthology.exit:Finished running handlers